L1 blob priority_gas_price runaway

[kubasiemion]

🐛 Bug Report

A rollup anchored on Holesky cannot publish blobs: the transaction fails as the blob_operator does not have 2000+ ETH (and increasing).

📝 Description

We run a rollup based on Holesky, with data publishing on L1.
After a period of operation with unfunded blob_operator, the transaction costs thousands of ETH due do the priority fee.
The transaction fails, but stays in the queue.
The rollup tries to submit again, with still higher priority fee, and guess what...
There is no obvious way to remove these transactions from the queue, they survive server restarts, so there is no (cheap) way to start submitting the blobs again:

🔄 Reproduction Steps

Start a rollup with unfunded blob operator

🤔 Expected Behavior

System should not offer premium gas price that it cannot afford
There should be a simple way to prune the tx queue.

🖥️ Environment

Ubuntu on AWS, stand-alone build from the repo head, ca #00badca

📎 Log Output

2024-12-10T16:11:32.851513Z  INFO zksync_node_api_server::web3::pubsub: Stop signal received, pubsub_block_notifier is shutting down
2024-12-10T16:11:32.966827Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Checking tx id: 1, operator_nonce: OperatorNonce { finalized: Nonce(0), latest: Nonce(0) }, tx nonce: 0
2024-12-10T16:11:32.970191Z  INFO EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Resending Blob tx 1 (nonce 0) at block 2909557 with base_fee_per_gas 6291456, priority_fee_per_gas 524288000000000, blob_fee_per_gas Some(524288), previously sent with base_fee_per_gas 3145728, priority_fee_per_gas 262144000000000, blob_fee_per_gas Some(262144), 
2024-12-10T16:11:33.130833Z  WARN EthTxManager::loop_iteration: zksync_eth_sender::eth_tx_manager: Error Sending Blob tx 1 (nonce 0) at block 2909557 with base_fee_per_gas 6291456, priority_fee_per_gas 524288000000000, blob_fee_per_gas Some(524288),error Ethereum gateway error: JSON-RPC request send_raw_transaction() failed: ErrorObject { code: ServerError(-32000), message: "insufficient funds for gas * price + value: balance 5000000000000000000, tx cost 7864320094440559476736, overshot 7859320094440559476736", data: None }