EVM transaction replacement #206

codchen · 2024-02-28T05:44:29Z

Describe your changes and provide context

Added replacement logic for EVM transactions that works as follows:

if a tx with the same nonce exists in the priority index that has a lower priority, the existing tx is replaced by the incoming tx
if a tx with the same nonce exists in the priority index that has an equal or higher priority, the incoming tx is dropped
After this change, there will not be any tx with duplicate nonce in the same queue inside the priority index, so the timestamp check in IsBefore is also removed.

As a requirement for keeping the invariant above, for any committed transaction, we will remove any tx in the mempool that has the same nonce regardless of whether it's the same tx or not. (For example, a node may have a tx of nonce-10 with content ABC in its local mempool, but received and executed a block proposal that contains a tx of nonce-10 with content DEF. In this case, we want to remove ABC from the local mempool after DEF's commit)

Testing performed to validate your change

unit test & local sei test

local sei test steps:

Block time -> 10s

block 1
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 1000 --from=admin --nonce=0 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 2000 --from=admin --nonce=0 --gas-fee-cap=1100000000000

block 1
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 1000 --from=admin --nonce=0 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 2000 --from=admin --nonce=0 --gas-fee-cap=900000000000

block 1
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 1000 --from=admin --nonce=0 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 2000 --from=admin --nonce=2 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 3000 --from=admin --nonce=2 --gas-fee-cap=1100000000000
block 2
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 10000 --from=admin --nonce=1 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 100000 --from=admin --nonce=3 --gas-fee-cap=1000000000000

block 1
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 1000 --from=admin --nonce=0 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 2000 --from=admin --nonce=2 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 3000 --from=admin --nonce=2 --gas-fee-cap=900000000000
block 2
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 10000 --from=admin --nonce=1 --gas-fee-cap=1000000000000
seid tx evm send 0x9f76969569764cFc33D597D0eb00242cDB11EAE2 100000 --from=admin --nonce=3 --gas-fee-cap=1000000000000

philipsu522

lgtm but let's wait for @stevenlanders to review

philipsu522 · 2024-03-01T16:15:20Z

internal/mempool/priority_queue.go

+				// should replace
+				// replace heap if applicable


nit comment fix

stevenlanders · 2024-03-05T16:17:19Z

internal/mempool/mempool.go

@@ -364,7 +364,7 @@ func (txmp *TxMempool) RemoveTxByKey(txKey types.TxKey) error {

 	// remove the committed transaction from the transaction store and indexes
 	if wtx := txmp.txStore.GetTxByHash(txKey); wtx != nil {
-		txmp.removeTx(wtx, false, true)
+		txmp.removeTx(wtx, false, true, true)


(once we have so many flags, it might make sense to split these out as named methods)

yeah makes sense. Or group them into a RemoveTxOptions struct that has named fields?

## Describe your changes and provide context See description in sei-protocol/sei-tendermint#206 ## Testing performed to validate your change --------- Co-authored-by: Steven Landers <[email protected]> Co-authored-by: Philip Su <[email protected]> Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Aleksandr Bezobchuk <[email protected]> Co-authored-by: Uday Patil <[email protected]>

* Make ReadMaxTxs atomic (#166) * Support pending transaction in mempool (#169) * fix unconfirmed tx to consider pending txs (#172) * fix pending pop (#173) * add TTL for pending txs (#174) * [EVM] Fix evm pending nonce (#179) * Perf: Increase buffer size for pubsub server to boost performance (#167) * Increase buffer size for pubsub server * Add more timeout for test failure * Add more timeout * Fix test split scripts * Fix test split * Fix unit test * Unit test * Unit test * [P2P] Optimize block pool requester retry and peer pick up logic (#170) * P2P Improvements: Fix block sync reactor and block pool retry logic * Revert "Add event data to result event (#165)" (#176) This reverts commit 72bb29c. * Fix block sync auto restart not working as expected (#175) * Fix edge case for blocksync (#178) * fix evm pending nonce * fix test * deflake a test * de-flake test * Revert "merge main" This reverts commit 58b9424, reversing changes made to 02d1478. * consider keep-in-cache logic when removing from cache * undo test tweaks --------- Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]> * Fix bug when popping pending TXs (#188) * Add mempool metrics for number of pending tx and expired txs (#189) * Add metrics for mempool pending transaction size * Add expired tx count metrics * [EVM] Allow multiple txs from same account in a block (#190) * add mempool prioritization with evm nonce * fix priority stability * index fixes * replace with binary search insert * impl binary search * fix removeTx to push next queued evm tx (#191) * fix expire metric (#193) * [EVM] Fix duplicate evm txs from priority queue (#195) * debug duplicate evm tx * add more logs * add some \ns * more logs * fix swap check * add-lockable-reap-by-gas * add invariant checks * fix invariant parenthesis * fix log * remove invalid invariant * fix nonce ordering pain * handle ordering of insert * fix remove * cleanup * fix imports * cleanup * avoid getTransactionByHash(hash) panic due to index * use Key() to compare instead of pointer * [EVM] prevent duplicate txs from getting inserted (#196) * prevent duplicates in mempool * use timestamp in priority queue * [EVM] Add logging for expiration (#198) * add logging for expired txs * cleanup * [EVM] Avoid returning nil transactions on ForEach (#197) * remove heapIndex to avoid nil scenario * avoid returning nil in loop (mimic Peek) * call callback from mempool (#200) * separate limit for pending tx (#202) * Add EVM txs eviction logic (#204) * Fix debug log (#205) * EVM transaction replacement (#206) (#208) * Add heapIndex with safety check (#213) * add heapIndex with safety check * cleanup * comment out for perf test * add back perf improvement * fix nil test * Use write-lock in (*TxPriorityQueue).ReapMax funcs (#209) ReapMaxBytesMaxGas and ReapMaxTxs funcs in TxPriorityQueue claim > Transactions returned are not removed from the mempool transaction > store or indexes. However, they use a priority queue to accomplish the claim > Transaction are retrieved in priority order. This is accomplished by popping all items out of the whole heap, and then pushing then back in sequentially. A copy of the heap cannot be obtained otherwise. Both of the mentioned functions use a read-lock (RLock) when doing this. This results in a potential scenario where multiple executions of the ReapMax can be started in parallel, and both would be popping items out of the priority queue. In practice, this can be abused by executing the `unconfirmed_txs` RPC call repeatedly. Based on our observations, running it multiple times per millisecond results in multiple threads picking it up at the same time. Such a scenario can be obtained via the WebSocket interface, and spamming `unconfirmed_txs` calls there. The behavior that happens is a `Panic in WSJSONRPC handler` when a queue item unexpectedly disappears for `mempool.(*TxPriorityQueue).Swap`. (`runtime error: index out of range [0] with length 0`) This can additionally lead to a `CONSENSUS FAILURE!!!` if the race condition occurs for `internal/consensus.(*State).finalizeCommit` when it tries to do `mempool.(*TxPriorityQueue).RemoveTx`, but the ReapMax has already removed all elements from the underlying heap. (`runtime error: index out of range [-1]`) This commit switches the lock type to a write-lock (Lock) to ensure no parallel modifications take place. This commit additionally updates the tests to allow parallel execution of the func calls in testing, as to prevent regressions (in case someone wants to downgrade the locks without considering the implications from the underlying heap usage). --------- Co-authored-by: Valters Jansons <[email protected]> * Pending Txs Update Condition (#214) * Add metrics for mempool size changes (#220) * [EVM] Adjust locking for replacement (#224) * Remove tx from cache when canAddPendingTx fails (#230) * add tx hash to evm info proto (#231) --------- Co-authored-by: codchen <[email protected]> Co-authored-by: Steven Landers <[email protected]> Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Jeremy Wei <[email protected]> Co-authored-by: Valters Jansons <[email protected]> Co-authored-by: Kartik Bhat <[email protected]>

## Describe your changes and provide context See description in sei-protocol/sei-tendermint#206 ## Testing performed to validate your change --------- Co-authored-by: Steven Landers <[email protected]> Co-authored-by: Philip Su <[email protected]> Co-authored-by: Yiming Zang <[email protected]> Co-authored-by: Aleksandr Bezobchuk <[email protected]> Co-authored-by: Uday Patil <[email protected]>

codchen requested review from philipsu522 and stevenlanders February 28, 2024 05:44

codchen mentioned this pull request Feb 28, 2024

Integrate with tendermint EVM tx replacement logic sei-protocol/sei-cosmos#446

Merged

EVM transaction replacement

bd9d427

codchen force-pushed the replacement-evm branch from 36ab01b to bd9d427 Compare February 28, 2024 05:57

codchen mentioned this pull request Feb 28, 2024

Integrate with tendermint EVM tx replacement logic sei-protocol/sei-chain#1394

Open

philipsu522 approved these changes Mar 1, 2024

View reviewed changes

stevenlanders approved these changes Mar 5, 2024

View reviewed changes

stevenlanders reviewed Mar 5, 2024

View reviewed changes

codchen merged commit f4e9c14 into evm Mar 7, 2024
23 checks passed

codchen added a commit that referenced this pull request Mar 7, 2024

EVM transaction replacement (#206)

a42599c

codchen added a commit that referenced this pull request Mar 7, 2024

EVM transaction replacement (#206) (#208)

77ad5bd

codchen mentioned this pull request Mar 7, 2024

Integrate with tendermint EVM tx replacement logic sei-protocol/sei-chain#1411

Merged

udpatil pushed a commit that referenced this pull request Mar 26, 2024

EVM transaction replacement (#206) (#208)

4af066a

udpatil pushed a commit that referenced this pull request Apr 16, 2024

EVM transaction replacement (#206) (#208)

21aa1cb

udpatil pushed a commit that referenced this pull request Apr 16, 2024

EVM transaction replacement (#206) (#208)

d90626d

udpatil pushed a commit that referenced this pull request Apr 19, 2024

EVM transaction replacement (#206) (#208)

14a366f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVM transaction replacement #206

EVM transaction replacement #206

codchen commented Feb 28, 2024

philipsu522 left a comment

philipsu522 Mar 1, 2024

stevenlanders Mar 5, 2024

codchen Mar 6, 2024

EVM transaction replacement #206

EVM transaction replacement #206

Conversation

codchen commented Feb 28, 2024

Describe your changes and provide context

Testing performed to validate your change

philipsu522 left a comment

Choose a reason for hiding this comment

philipsu522 Mar 1, 2024

Choose a reason for hiding this comment

stevenlanders Mar 5, 2024

Choose a reason for hiding this comment

codchen Mar 6, 2024

Choose a reason for hiding this comment