[Integration test] v0.8.0-rc6 non-determinism (mainnet)

[gusiri]

version: v0.8.0-rc6
network: mainnet
besu node: devnet besu_load_florian_1 (private IP: 10.40.41.87)

lt files: https://drive.google.com/file/d/1hUHoWgK2meVaAVPea4wrPafevNjEiuLr/view?usp=drive_link

Problem

• The mainnet Besu node besu_load_florian_1 (private IP: 10.40.41.87) shows non-determinism for the ranges listed below.
• Traces for these ranges fail when generated by this specific Besu node but pass when generated by other healthy Besu nodes.
• When I split the range into smaller pieces (e.g., single blocks), they all pass, even if they were generated by the problematic Besu node. This behavior is exactly the same as the non-determinism issue we resolved in the past.

Related Issues

• Non-determinism issue resolved in the past: non-determinism - mainnet v0.6.0-rc8 (txndata error) #1316
• Non determinism issue was caused by different configurations; --Xevm-worldstate-update-mode=JOURNALED should not be JOURNALED. (--Xevm-worldstate-update-mode=JOURNALED interfers with tracing linea-besu#47)

Constraints Errors

Type 1.1

• txndata.cumulative-gas---update-at-transaction-threshold
• <prelude>.loginfo-into-hub
• txndata.setting-refund-effective

2555570-2555577 (.lt file attached)

Type 1.2

• txndata.cumulative-gas---update-at-transaction-threshold
• txndata.setting-refund-effective

Note: No <prelude>.loginfo-into-hub error in this range.

2555570-2555576 (.lt file attached)

Type 2

• txndata.setting-refund-effective

7680466-7680469 (.lt file attached) 
2648332-2648371
3280602-3280609
3506963-3507013
3734888-3734962
3985691-3985793
4385302-4385370
4817239-4817296

[OlivierBBB]

The presence of <prelude>.loginfo-into-hub, besides the <prelude> of which I don't know what it means, is surprising. There should be no mention of the HUB anywhere.

[DavePearce]

@OlivierBBB <prelude> refers to the top-level module. Often, lookups are defined in the top-level module (as I guess is true in this case).

[DavePearce]

For the range 7680466-7680469 we have this report from go-corset:

ERRO[0004] constraint "txndata.setting-refund-effective" does not hold (row 283) (HIR)
failing constraint txndata.setting-refund-effective:
                  |    280 |    281 |    282 |              283 |              284 |
       ABS_TX_NUM |   0x20 |   0x20 |   0x20 |             0x21 |             0x21 |
     GAS_LEFTOVER | 0x845d | 0x845d | 0x845d |          0x114ce |          0x114ce |
   REFUND_COUNTER |    0x0 |    0x0 |    0x0 | 0xffffffffffff.. | 0xffffffffffff.. |
 REFUND_EFFECTIVE | 0x845d | 0x845d | 0x845d |          0x1020e |          0x1020e |
              RES |    0x1 |    0x1 |    0x1 |              0x1 |              0x1 |

Also, for the txndata module there are 384 rows. So, this constraint is not failing on the last row, for example.

@OlivierBBB It looks like the refund counter has gone negative?

[DavePearce]

Here is a wider snapshot of the affected area in txndata:

                   |               #283 |               #284 |               #285 |               #286 |               #287 |               #288 |               #289 |               #290 |               #291 |               #292 |               #293 |               #294 |
 txndata.ABS_TX_NUM |                 21 |                 21 |                 21 |                 21 |                 21 |                 21 |                 21 |                 22 |                 22 |                 22 |                 22 |                 22 |
 txndata.ABS_TX_N.. |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |                 2c |
 txndata.ARG_ONE_LO |     1761a3a717eb60 |               5248 |               6232 |   ffffffffffffed40 |                  4 |                  7 |                  0 |               f212 |        e09446ed180 |               55fc |               9459 |                  0 |
 txndata.ARG_TWO_LO |     1ad8dbc677a84c |              17700 |                  5 |               13a3 |                  0 |            444ae90 |                  0 |   ffffffffffffffff |   41591120b69cfe19 |              35f30 |                  5 |               1dab |
    txndata.BASEFEE |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |                  7 |
 txndata.BLOCK_GA.. |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |            3a2c940 |
 txndata.CALL_DAT.. |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                 c4 |                 c4 |                 c4 |                 c4 |                 c4 |
 txndata.CODE_FRA.. |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
 txndata.COINBASE.. |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
 txndata.COINBASE.. |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
  txndata.COPY_TXCD |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |
         txndata.CT |                  1 |                  2 |                  3 |                  4 |                  5 |                  6 |                  7 |                  0 |                  1 |                  2 |                  3 |                  4 |
   txndata.EUC_FLAG |                  0 |                  0 |                  1 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  1 |                  0 |
    txndata.FROM_HI |           1974f7e5 |           1974f7e5 |           1974f7e5 |           1974f7e5 |           1974f7e5 |           1974f7e5 |           1974f7e5 |           339d413c |           339d413c |           339d413c |           339d413c |           339d413c |
    txndata.FROM_LO | c4d7d3795b990ba5.. | c4d7d3795b990ba5.. | c4d7d3795b990ba5.. | c4d7d3795b990ba5.. | c4d7d3795b990ba5.. | c4d7d3795b990ba5.. | c4d7d3795b990ba5.. | cefd986b1b3647a9.. | cefd986b1b3647a9.. | cefd986b1b3647a9.. | cefd986b1b3647a9.. | cefd986b1b3647a9.. |
 txndata.GAS_CUMU.. |               74f2 |               74f2 |               74f2 |               74f2 |               74f2 |               74f2 |               74f2 |              1094b |              1094b |              1094b |              1094b |              1094b |
 txndata.GAS_INIT.. |              124b8 |              124b8 |              124b8 |              124b8 |              124b8 |              124b8 |              124b8 |              30934 |              30934 |              30934 |              30934 |              30934 |
 txndata.GAS_LEFT.. |              114ce |              114ce |              114ce |              114ce |              114ce |              114ce |              114ce |              2cad7 |              2cad7 |              2cad7 |              2cad7 |              2cad7 |
  txndata.GAS_LIMIT |              17700 |              17700 |              17700 |              17700 |              17700 |              17700 |              17700 |              35f30 |              35f30 |              35f30 |              35f30 |              35f30 |
  txndata.GAS_PRICE |            444ae90 |            444ae90 |            444ae90 |            444ae90 |            444ae90 |            444ae90 |            444ae90 |            429ac88 |            429ac88 |            429ac88 |            429ac88 |            429ac88 |
 txndata.INIT_COD.. |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
 txndata.INITIAL_.. |     1ad8dbc677a84c |     1ad8dbc677a84c |     1ad8dbc677a84c |     1ad8dbc677a84c |     1ad8dbc677a84c |     1ad8dbc677a84c |     1ad8dbc677a84c |   41591120b69cfe19 |   41591120b69cfe19 |   41591120b69cfe19 |   41591120b69cfe19 |   41591120b69cfe19 |
       txndata.INST |                  f |                  f |                  0 |                 10 |                 15 |                  f |                  0 |                 10 |                  f |                  f |                  0 |                 10 |
     txndata.IS_DEP |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
 txndata.IS_LAST_.. |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
      txndata.NONCE |                 4a |                 4a |                 4a |                 4a |                 4a |                 4a |                 4a |               f212 |               f212 |               f212 |               f212 |               f212 |
 txndata.OUTGOING.. |           e5d7c2a4 |                  0 |                  0 |                 40 |                  0 |                  0 |                  0 |                  0 |           3c226981 |                  0 |                  0 |                3f4 |
 txndata.OUTGOING.. | 4ffddf6b295a15c1.. |                 4a |     175b630b62fb60 |                  4 |              17700 |            444ae90 |                  0 |                  0 | 1836af69497e5f48.. |               f212 |                  0 |                 c4 |
 txndata.OUTGOING.. |                  1 |               74f2 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  1 |              1094b |                  0 |                  0 |
 txndata.PHASE_RL.. |                  8 |                  3 |                  9 |                  a |                  7 |                  4 |                  f |                  1 |                  8 |                  3 |                  9 |                  a |
 txndata.PHASE_RL.. |                  2 |                  3 |                  0 |                  0 |                  0 |                  0 |                  0 |                  7 |                  2 |                  3 |                  0 |                  0 |
 txndata.PRIORITY.. |            444ae89 |            444ae89 |            444ae89 |            444ae89 |            444ae89 |            444ae89 |            444ae89 |            429ac81 |            429ac81 |            429ac81 |            429ac81 |            429ac81 |
 txndata.REFUND_C.. |   ffffffffffffed40 |   ffffffffffffed40 |   ffffffffffffed40 |   ffffffffffffed40 |   ffffffffffffed40 |   ffffffffffffed40 |   ffffffffffffed40 |                  0 |                  0 |                  0 |                  0 |                  0 |
 txndata.REFUND_E.. |              1020e |              1020e |              1020e |              1020e |              1020e |              1020e |              1020e |              2cad7 |              2cad7 |              2cad7 |              2cad7 |              2cad7 |
  txndata.REL_BLOCK |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |                  4 |
 txndata.REL_TX_NUM |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  2 |                  2 |                  2 |                  2 |                  2 |
 txndata.REL_TX_N.. |                  c |                  c |                  c |                  c |                  c |                  c |                  c |                  c |                  c |                  c |                  c |                  c |
 txndata.REQUIRES.. |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |
        txndata.RES |                  1 |                  1 |               13a3 |                  0 |                  0 |                  1 |                  0 |                  1 |                  1 |                  1 |               1dab |                  1 |
 txndata.STATUS_C.. |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |
      txndata.TO_HI |           e5d7c2a4 |           e5d7c2a4 |           e5d7c2a4 |           e5d7c2a4 |           e5d7c2a4 |           e5d7c2a4 |           e5d7c2a4 |           3c226981 |           3c226981 |           3c226981 |           3c226981 |           3c226981 |
      txndata.TO_LO | 4ffddf6b295a15c1.. | 4ffddf6b295a15c1.. | 4ffddf6b295a15c1.. | 4ffddf6b295a15c1.. | 4ffddf6b295a15c1.. | 4ffddf6b295a15c1.. | 4ffddf6b295a15c1.. | 1836af69497e5f48.. | 1836af69497e5f48.. | 1836af69497e5f48.. | 1836af69497e5f48.. | 1836af69497e5f48.. |
      txndata.TYPE0 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |                  1 |
      txndata.TYPE1 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
      txndata.TYPE2 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |                  0 |
      txndata.VALUE |     175b630b62fb60 |     175b630b62fb60 |     175b630b62fb60 |     175b630b62fb60 |     175b630b62fb60 |     175b630b62fb60 |     175b630b62fb60 |                  0 |                  0 |                  0 |                  0 |                  0 |
   txndata.WCP_FLAG |                  1 |                  1 |                  0 |                  1 |                  1 |                  1 |                  0 |                  1 |                  1 |                  1 |                  0 |                  1 

[OlivierBBB]

@DavePearce Is this the end of the trace ? Or somewhere in the middle ?

[OlivierBBB]

@DavePearce Yes, it looks like the REFUND_COUNTER is at fault ...

[DavePearce]

I make this calculation:

• REFUND_COUNTER=-4800 (roughly I think)
• REFUND_EFFECTIVE=66062
• GET_FULL_REFUND=0
• REFUND_LIMIT=0
• GAS_LEFTOVER=70862

So, constraints fails because REFUND_EFFECTIVE != GAS_LEFTOVER

BUT: I think that REFUND_EFFECTIVE + REFUND_COUNTER == GAS_LEFTOVER. So that suggests get_full_refund should be non-zero.

[DavePearce]

Is this the end of the trace ? Or somewhere in the middle ?

@OlivierBBB Its at row 283 out of 384 rows. So, towards the end.

[FlorianHuc]

besu node started with --Xevm-worldstate-update-mode=JOURNALED

I update the start entry to test, it would mean the previous fix was not enough.

[gusiri]

I think that fixed this issue.

Type 1

2555570-2555577 (passed)
2555570-2555576 (passed)

Type 2

7680466-7680469 (passed)

I am still checking the other ranges.

[gusiri]

Fixed by hyperledger/besu#7758