Issues starting a teku / reth node on sepolia

[rolfyone]

difficult to start teku / reth pair on sepolia due to the following error:

Unsure if it happens on other EL / CL, but does consistently apparently occur with teku / reth.

2024-12-09 05:21:13.723 INFO  - Quartz scheduler 'QuartzScheduler' initialized from an externally provided properties instance.
2024-12-09 05:21:13.723 INFO  - Quartz scheduler version: 2.3.2
2024-12-09 05:21:13.758 INFO  - Initializing storage
2024-12-09 05:21:14.118 INFO  - Storage initialization complete
2024-12-09 05:21:14.119 WARN  - Ignoring weak subjectivity period check (--ignore-weak-subjectivity-period-enabled). Syncing from outside of the weak subjectivity period is considered UNSAFE.
2024-12-09 05:21:14.134 INFO  - Loaded C-KZG-4844 library
2024-12-09 05:21:16.408 INFO  - BLS: loaded BLST library
2024-12-09 05:21:16.660 INFO  - Completed regeneration of state for block 0x42a176d41c1f497dcfe9a47d58ec5a78f1c787ba87dc1428afc7dcf46b6f82ed at slot 6496512 by replaying 30 blocks. Took 418ms
2024-12-09 05:21:16.858 INFO  - Execution Client is online
2024-12-09 05:21:16.892 INFO  - Loading generated p2p private key from: generated-node-key
2024-12-09 05:21:16.897 FATAL - Exiting due to fatal error in fork choice
java.util.concurrent.CompletionException: tech.pegasys.teku.infrastructure.exceptions.FatalServiceFailureException: Optimistic sync finalized an invalid transition block. Unable to recover.
at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:147) ~[teku-infrastructure-async-24.10.3.jar:24.10.3]
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]
Caused by: tech.pegasys.teku.infrastructure.exceptions.FatalServiceFailureException: Optimistic sync finalized an invalid transition block. Unable to recover.
at tech.pegasys.teku.statetransition.forkchoice.MergeTransitionBlockValidator.lambda$verifyFinalizedTransitionBlock$2(MergeTransitionBlockValidator.java:100) ~[teku-ethereum-statetransition-24.10.3.jar:24.10.3]
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:147) ~[teku-infrastructure-async-24.10.3.jar:24.10.3]
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]

Causing a customer significant issues, will put to top of current for investigation.

[zilm13]

Provided Sepolia Reth contains incorrect totalDifficulty calculated for preMerge block:

curl --request POST \
     --url http://127.0.0.1:8545/ \
     --header 'accept: application/json' \
     --header 'content-type: application/json' \
     --data '
{
  "id": 1,
  "jsonrpc": "2.0",
  "method": "eth_getBlockByHash",
  "params": [
    "0xa4f33f2b549616b022927d13dfa38bfb5b393f72a59d0610b6ffdd8bc6c3bcb9",
    false
  ]
}
'
{
   "jsonrpc":"2.0",
   "id":1,
   "result":{
      "hash":"0xa4f33f2b549616b022927d13dfa38bfb5b393f72a59d0610b6ffdd8bc6c3bcb9",
      "parentHash":"0xf55ce9cb54382b2a9733aff3ef8c10f202cacfbb02e113da9cfa6992c9bfa8a1",
      "sha3Uncles":"0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
      "miner":"0xedf0f83b6da795931f668e64d82eb8c5aad46340",
      "stateRoot":"0x11a26d0b32d1456838f123bf7d28b013c5f57c5f93c2c1c159a3b4dd14d0a554",
      "transactionsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
      "receiptsRoot":"0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
      "logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "difficulty":"0x25a9d6e4dd",
      "number":"0x1621a7",
      "gasLimit":"0x1c9c380",
      "gasUsed":"0x0",
      "timestamp":"0x62c594e2",
      "extraData":"0xd883010a14846765746888676f312e31382e33856c696e7578",
      "mixHash":"0x549529ee91f36bcdd06b5cc9df9d8ee2a4ca4f02542dfb550273ac439c60b41f",
      "nonce":"0xcb7f36a16b0dbcdb",
      "baseFeePerGas":"0x7",
      "totalDifficulty":"0x3c656d23029ab0",
      "size":"0x21f",
      "uncles":[
         
      ],
      "transactions":[
         
      ]
   }
}

0x3c656d23029ab0 == 17,000,018,015,853,232
while correct total difficulty for this block is 16,999,856,173,644,535 . We cannot ask Reth to fix this as total difficulty is going to be deprecated and was removed from Reth a week ago paradigmxyz/reth#13303, so #8942 was created