Reth has trouble sending Payload Status to lighthouse

[ChosunOne]

Describe the bug

This is a companion issue to the one reported here: sigp/lighthouse#6734

Either Lighthouse or Reth are struggling to communicate payload status with one another fairly regularly. I'm not certain if it is every block, but it happens frequently enough so that I can see it every epoch or so. I'm running everything on the same machine, so connectivity isn't the problem.

Steps to reproduce

Rust:

% rustc --version
rustc 1.83.0 (90b35a623 2024-11-26)

Lighthouse:

% lighthouse --version
Lighthouse v6.0.1-0d90135
BLS library: blst
BLS hardware acceleration: true
SHA256 hardware acceleration: true
Allocator: jemalloc
Profile: maxperf
Specs: mainnet (true), minimal (false), gnosis (false)

Reth:

% reth --version
reth Version: 1.1.4-dev
Commit SHA: 058cfe2f98952633da53f877b538a1ea3f9d657f
Build Timestamp: 2024-12-19T15:21:06.089859378Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

Node logs

Dec 20 02:29:02 viserion reth[69389]: 2024-12-19T17:29:02.827294Z  INFO Block added to canonical chain number=21438062 hash=0x91a41a83e762ffe5b19a44082b2f05d3afd6ec6abebff1c7efac83e1263dd62d peers=87 txs=159 gas=13.97 Mgas gas_throughput=9.80 Mgas/second full=46.6% base_fee=21.55gwei blobs=0 excess_blobs=631 elapsed=1.424893523s
Dec 20 02:29:03 viserion reth[69389]: 2024-12-19T17:29:03.011123Z  INFO Canonical chain committed number=21438062 hash=0x91a41a83e762ffe5b19a44082b2f05d3afd6ec6abebff1c7efac83e1263dd62d elapsed=1.128555ms
Dec 20 02:29:14 viserion reth[69389]: 2024-12-19T17:29:14.967405Z  INFO Block added to canonical chain number=21438063 hash=0xf978ecc3ab8118ebf12bd9d1da31093b81970174ee2a944b81afb128d220986d peers=86 txs=201 gas=15.95 Mgas gas_throughput=9.21 Mgas/second full=53.2% base_fee=21.37gwei blobs=6 excess_blobs=628 elapsed=1.730926651s
Dec 20 02:29:15 viserion reth[69389]: 2024-12-19T17:29:15.147110Z  INFO Canonical chain committed number=21438063 hash=0xf978ecc3ab8118ebf12bd9d1da31093b81970174ee2a944b81afb128d220986d elapsed=570.289µs
Dec 20 02:29:30 viserion reth[69389]: 2024-12-19T17:29:30.126923Z  INFO Block added to canonical chain number=21438064 hash=0x7b36feddbffc5a17bfa2ff13d062ce0da6d57c5caf82f12132d20ec7a4aa2591 peers=84 txs=146 gas=15.15 Mgas gas_throughput=3.14 Mgas/second full=50.5% base_fee=21.54gwei blobs=0 excess_blobs=631 elapsed=4.829080736s
Dec 20 02:29:30 viserion reth[69389]: 2024-12-19T17:29:30.321637Z  INFO Canonical chain committed number=21438064 hash=0x7b36feddbffc5a17bfa2ff13d062ce0da6d57c5caf82f12132d20ec7a4aa2591 elapsed=633.349µs
Dec 20 02:29:40 viserion reth[69389]: 2024-12-19T17:29:40.027004Z  INFO Block added to canonical chain number=21438065 hash=0xffa7839205d3442b05a8d48ebee0b4ba2d3bf2a43b91f3f371b6499b87d9857d peers=83 txs=233 gas=20.10 Mgas gas_throughput=9.59 Mgas/second full=67.0% base_fee=21.56gwei blobs=5 excess_blobs=628 elapsed=2.096544851s
Dec 20 02:29:40 viserion reth[69389]: 2024-12-19T17:29:40.246623Z  INFO Canonical chain committed number=21438065 hash=0xffa7839205d3442b05a8d48ebee0b4ba2d3bf2a43b91f3f371b6499b87d9857d elapsed=743.437µs
Dec 20 02:29:52 viserion reth[69389]: 2024-12-19T17:29:52.345813Z  INFO Block added to canonical chain number=21438066 hash=0xc47471c4a66b7e5f540d13a462e041dc6370ade96087bfd9a3043efab41392a0 peers=81 txs=158 gas=12.96 Mgas gas_throughput=6.98 Mgas/second full=43.2% base_fee=22.48gwei blobs=6 excess_blobs=630 elapsed=1.856591584s
Dec 20 02:29:52 viserion reth[69389]: 2024-12-19T17:29:52.556883Z  INFO Canonical chain committed number=21438066 hash=0xc47471c4a66b7e5f540d13a462e041dc6370ade96087bfd9a3043efab41392a0 elapsed=2.108119ms
Dec 20 02:30:02 viserion reth[69389]: 2024-12-19T17:30:02.380929Z  INFO Block added to canonical chain number=21438067 hash=0x60ed62f2b5cc97a7352141f2967d25aa9fab91b444e02353f9bcb965c0d975f2 peers=82 txs=125 gas=10.07 Mgas gas_throughput=8.66 Mgas/second full=33.6% base_fee=22.09gwei blobs=0 excess_blobs=633 elapsed=1.162100465s
Dec 20 02:30:02 viserion reth[69389]: 2024-12-19T17:30:02.583355Z  INFO Canonical chain committed number=21438067 hash=0x60ed62f2b5cc97a7352141f2967d25aa9fab91b444e02353f9bcb965c0d975f2 elapsed=447.547µs
Dec 20 02:30:15 viserion reth[69389]: 2024-12-19T17:30:15.482908Z  INFO Block added to canonical chain number=21438068 hash=0x7ac353a26b609b69c9c2aa832db4d64b96914eecf306318d6b07a6e48b3231d2 peers=83 txs=238 gas=17.60 Mgas gas_throughput=6.84 Mgas/second full=58.7% base_fee=21.19gwei blobs=3 excess_blobs=630 elapsed=2.572723124s
Dec 20 02:30:15 viserion reth[69389]: 2024-12-19T17:30:15.697971Z  INFO Canonical chain committed number=21438068 hash=0x7ac353a26b609b69c9c2aa832db4d64b96914eecf306318d6b07a6e48b3231d2 elapsed=862.393µs
Dec 20 02:30:33 viserion reth[69389]: 2024-12-19T17:30:33.725088Z  INFO Block added to canonical chain number=21438069 hash=0xe59075be1edf60376438c2352e5b68ba19cd78ae5e5636539b77d21dabdcef14 peers=83 txs=187 gas=14.24 Mgas gas_throughput=1.61 Mgas/second full=47.5% base_fee=21.65gwei blobs=2 excess_blobs=630 elapsed=8.86600547s
Dec 20 02:30:33 viserion reth[69389]: 2024-12-19T17:30:33.727088Z ERROR Failed to send event: Ok(PayloadStatus { status: Valid, latest_valid_hash: Some(0xe59075be1edf60376438c2352e5b68ba19cd78ae5e5636539b77d21dabdcef14) })

Platform(s)

Linux (x86)

Container Type

Not running in a container

What version/commit are you on?

reth Version: 1.1.4-dev
Commit SHA: 058cfe2
Build Timestamp: 2024-12-19T15:21:06.089859378Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

What database version are you on?

Current database version: 2
Local database version: 2

Which chain / network are you on?

mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

block_interval = 5

If you've built Reth from source, provide the full command you used

RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf --features jemalloc,asm-keccak

Code of Conduct

• I agree to follow the Code of Conduct