How does the reth import command work? [error: file client has no tip]

[debjit-bw]

We have imported a state up to a block (say 1000, without evm). Now I want to import a series of blocks (say, from 1001 to 2000). I try to use the reth import command, but it fails, with the file client has no tip error. I want to know more about when that error occurs? Here's what I'm doing:

Manually creating the block rlp (RLP encoding the [block header (eip 1559 format), [RLP encoded txs], ] array) and concatenating the serialized bytes (bytes of block 1001, bytes of block 1002...) and writing to the file (which i'm trying to import). Is that for format it expects?

Here's the full logs for context:

[interesting parts]

i'll refer to block 26479201 as just 201

downloaders::bodies: Downloading bodies count=2 range=26479201..=26479202 (why? the file has 10 blocks, 201 to 210)
DEBUG reth::cli: Tip manually set tip=0xcb4a817a56e247112339ad3e0e8afc9f4628b93d2a26071e7f8a2ec3b2db8170 (which is hash of block 202)
INFO sync::stages::headers: Received headers total=2 from_block=26479202 to_block=26479201
INFO sync::stages::headers: Writing headers total=2
INFO sync::stages::headers: Writing headers hash index total=2
DEBUG downloaders::file: new bytes were read from file max_chunk_byte_len=1000000000 prev_read_bytes_len=5085 new_read_bytes_target_len=0 new_read_bytes_len=0 next_chunk_byte_len=5085 remaining_file_byte_len=0
INFO reth::cli: Importing chain file chunk
ERROR reth::cli: Error: file client has no tip

[full]

debjit@MacBookPro reth_gnosis % ./target/debug/reth import \
  -vvvv \
  --chain /Users/debjit/Drawer/gnosis/reth_gnosis/scripts/mainnet_post_merge.json \
  --no-state /Users/debjit/Drawer/gnosis/rlp/concatenated_headers_26479201_26479210.rlp    

2024-12-10T16:39:40.288945Z  INFO reth::cli: Importing with custom cli
2024-12-10T16:39:40.297385Z  INFO reth::cli: reth 1.1.2-dev (8f61af01) starting
2024-12-10T16:39:40.297620Z  INFO reth::cli: Disabled stages requiring state
2024-12-10T16:39:40.297917Z DEBUG reth::cli: Chunking chain import chunk_byte_len=1000000000
2024-12-10T16:39:40.314874Z  INFO reth::cli: Opening storage db_path="/Users/debjit/Library/Application Support/reth/xdai/db" sf_path="/Users/debjit/Library/Application Support/reth/xdai/static_files"
2024-12-10T16:39:40.356499Z  INFO reth::cli: Verifying storage consistency.
2024-12-10T16:39:40.452013Z DEBUG reth::cli: Initializing genesis chain=xdai genesis=0x347dccceb9833c1e433d1d5153a479eb5a1041bb07cddd97853a74a086e56eb6
2024-12-10T16:39:40.454547Z DEBUG reth_db_common::init: Genesis already written, skipping.
2024-12-10T16:39:40.454924Z  INFO reth::cli: Consensus engine initialized
2024-12-10T16:39:40.460125Z DEBUG downloaders::file: new bytes were read from file max_chunk_byte_len=1000000000 prev_read_bytes_len=0 new_read_bytes_target_len=7272 new_read_bytes_len=7272 next_chunk_byte_len=7272 remaining_file_byte_len=0
2024-12-10T16:39:40.464382Z  INFO reth::cli: Importing chain file chunk
2024-12-10T16:39:40.465617Z  INFO reth::cli: Chain file chunk read
2024-12-10T16:39:40.474533Z  INFO downloaders::bodies: Downloading bodies count=2 range=26479201..=26479202
2024-12-10T16:39:40.480749Z DEBUG reth::cli: Tip manually set tip=0xcb4a817a56e247112339ad3e0e8afc9f4628b93d2a26071e7f8a2ec3b2db8170
2024-12-10T16:39:40.481781Z  INFO reth::cli: Starting sync pipeline
2024-12-10T16:39:40.484370Z DEBUG static_file: StaticFileProducer started targets=StaticFileTargets { headers: Some(26478651..=26479200), receipts: Some(26478651..=26479200), transactions: Some(26478651..=26479200) }
2024-12-10T16:39:40.487679Z DEBUG static_file: StaticFileProducer segment segment=Receipts block_range=26478651..=26479200
2024-12-10T16:39:40.487678Z DEBUG static_file: StaticFileProducer segment segment=Transactions block_range=26478651..=26479200
2024-12-10T16:39:40.487682Z DEBUG static_file: StaticFileProducer segment segment=Headers block_range=26478651..=26479200
2024-12-10T16:39:40.544336Z DEBUG static_file: Finished StaticFileProducer segment segment=Headers block_range=26478651..=26479200 elapsed=56.185416ms
2024-12-10T16:39:40.557191Z DEBUG static_file: Finished StaticFileProducer segment segment=Transactions block_range=26478651..=26479200 elapsed=69.400166ms
2024-12-10T16:39:40.589866Z DEBUG static_file: Finished StaticFileProducer segment segment=Receipts block_range=26478651..=26479200 elapsed=102.09475ms
2024-12-10T16:39:40.611446Z DEBUG provider::static_file: Commit segment=Headers path="/Users/debjit/Library/Application Support/reth/xdai/static_files/static_file_headers_26000000_26499999" duration=21.140625ms
2024-12-10T16:39:40.632404Z DEBUG provider::static_file: Commit segment=Transactions path="/Users/debjit/Library/Application Support/reth/xdai/static_files/static_file_transactions_26000000_26499999" duration=20.490917ms
2024-12-10T16:39:40.653441Z DEBUG provider::static_file: Commit segment=Receipts path="/Users/debjit/Library/Application Support/reth/xdai/static_files/static_file_receipts_26000000_26499999" duration=20.618875ms
2024-12-10T16:39:40.656141Z DEBUG static_file: StaticFileProducer finished targets=StaticFileTargets { headers: Some(26478651..=26479200), receipts: Some(26478651..=26479200), transactions: Some(26478651..=26479200) } elapsed=171.4515ms
2024-12-10T16:39:40.657283Z DEBUG pruner: Pruner started tip_block_number=26479200
2024-12-10T16:39:40.657506Z DEBUG pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=26479200 prune_mode=Before(26479201)
2024-12-10T16:39:40.663993Z DEBUG pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=26479200 prune_mode=Before(26479201) segment_output.pruned=1653
2024-12-10T16:39:40.665172Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=26479200 prune_mode=Before(26479201)
2024-12-10T16:39:40.671545Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=26479200 prune_mode=Before(26479201) segment_output.pruned=1218
2024-12-10T16:39:40.671645Z DEBUG pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=26479200 prune_mode=Before(26479201)
2024-12-10T16:39:40.681596Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=26479200 prune_mode=Before(26479201) segment_output.pruned=1218
2024-12-10T16:39:40.681631Z DEBUG pruner: Pruner finished tip_block_number=26479200 elapsed=24.324917ms deleted_entries=4089 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 18446744073709551615, deleted: 4089 }), time_limit: None } output=PrunerOutput { progress: Finished, segments: [(Headers, SegmentOutput { progress: Finished, pruned: 1653, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(26479200), tx_number: None }) }), (Transactions, SegmentOutput { progress: Finished, pruned: 1218, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(26479200), tx_number: Some(1217) }) }), (Receipts, SegmentOutput { progress: Finished, pruned: 1218, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(26479200), tx_number: Some(1217) }) })] } stats=[PrunedSegmentInfo { segment: Headers, pruned: 1653, progress: Finished }, PrunedSegmentInfo { segment: Transactions, pruned: 1218, progress: Finished }, PrunedSegmentInfo { segment: Receipts, pruned: 1218, progress: Finished }]
2024-12-10T16:39:40.874057Z DEBUG storage::db::mdbx: Commit total_duration=191.972417ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 6, audit: 0, write: 0, sync: 12564, ending: 0, whole: 12572, gc_cputime: 220, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 14, work_xtime_cpu: 1087, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-12-10T16:39:40.875759Z DEBUG sync::stages::headers: Commencing sync tip=Hash(0xcb4a817a56e247112339ad3e0e8afc9f4628b93d2a26071e7f8a2ec3b2db8170) head=0x01c459804cf019eb350a3c0adef00e1ef4d6247e90a2298211142faa04a5116b
2024-12-10T16:39:40.876057Z  INFO sync::stages::headers: Received headers total=2 from_block=26479202 to_block=26479201
2024-12-10T16:39:40.876690Z  INFO reth_node_events::node: Preparing stage pipeline_stages=1/5 stage=Headers checkpoint=26479200 target=26479202
2024-12-10T16:39:40.876979Z  INFO sync::stages::headers: Writing headers total=2
2024-12-10T16:39:40.876991Z  INFO reth_node_events::node: Executing stage pipeline_stages=1/5 stage=Headers checkpoint=26479200 target=26479202
2024-12-10T16:39:40.878838Z  INFO sync::stages::headers: Writing headers hash index total=2
2024-12-10T16:39:40.880792Z  INFO reth_node_events::node: Finished stage pipeline_stages=1/5 stage=Headers checkpoint=26479202 target=26479202 stage_progress=100.00%
2024-12-10T16:39:40.901058Z DEBUG provider::static_file: Commit segment=Headers path="/Users/debjit/Library/Application Support/reth/xdai/static_files/static_file_headers_26000000_26499999" duration=20.278959ms
2024-12-10T16:39:41.090898Z DEBUG storage::db::mdbx: Commit total_duration=189.56ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 4, audit: 0, write: 0, sync: 12414, ending: 0, whole: 12419, gc_cputime: 167, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 3, work_xtime_cpu: 403, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 113, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-12-10T16:39:41.091143Z  INFO reth_node_events::node: Preparing stage pipeline_stages=2/5 stage=Bodies checkpoint=26479200 target=26479202
2024-12-10T16:39:41.095061Z  INFO reth_node_events::node: Executing stage pipeline_stages=2/5 stage=Bodies checkpoint=26479200 target=26479202
2024-12-10T16:39:41.095619Z DEBUG sync::stages::bodies: Commencing sync stage_progress=26479201 target=26479202
2024-12-10T16:39:41.098873Z DEBUG providers::db: Inserted block body block_number=26479201 actions=[(InsertBlockBodyIndices, 333.208µs), (InsertTransactionBlocks, 63.417µs)]
2024-12-10T16:39:41.098997Z DEBUG providers::db: Inserted block body block_number=26479202 actions=[(InsertBlockBodyIndices, 16.959µs), (InsertTransactionBlocks, 3.583µs)]
2024-12-10T16:39:41.099195Z  INFO reth_node_events::node: Finished stage pipeline_stages=2/5 stage=Bodies checkpoint=26479202 target=26479202 stage_progress=0.00%
2024-12-10T16:39:41.119070Z DEBUG provider::static_file: Commit segment=Transactions path="/Users/debjit/Library/Application Support/reth/xdai/static_files/static_file_transactions_26000000_26499999" duration=19.911334ms
2024-12-10T16:39:41.308809Z DEBUG storage::db::mdbx: Commit total_duration=189.466125ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 4, audit: 0, write: 0, sync: 12407, ending: 0, whole: 12412, gc_cputime: 172, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 2, work_xtime_cpu: 315, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 102, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-12-10T16:39:41.309091Z  INFO reth_node_events::node: Preparing stage pipeline_stages=3/5 stage=SenderRecovery checkpoint=26479200 target=26479202
2024-12-10T16:39:41.309326Z  INFO reth_node_events::node: Executing stage pipeline_stages=3/5 stage=SenderRecovery checkpoint=26479200 target=26479202
2024-12-10T16:39:41.309343Z  INFO sync::stages::sender_recovery: Recovering senders tx_range=1218..1224
2024-12-10T16:39:41.310203Z DEBUG sync::stages::sender_recovery: Sending batch for processing tx_range=1218..1224
2024-12-10T16:39:41.310524Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1218..1224
2024-12-10T16:39:41.320531Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1218..1224
2024-12-10T16:39:41.320775Z  INFO reth_node_events::node: Finished stage pipeline_stages=3/5 stage=SenderRecovery checkpoint=26479202 target=26479202 stage_progress=0.49%
2024-12-10T16:39:41.514207Z DEBUG storage::db::mdbx: Commit total_duration=193.453083ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 4, audit: 0, write: 0, sync: 12668, ending: 0, whole: 12673, gc_cputime: 180, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 2, work_xtime_cpu: 266, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 132, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-12-10T16:39:41.514467Z  INFO reth_node_events::node: Preparing stage pipeline_stages=4/5 stage=TransactionLookup checkpoint=26479200 target=26479202
2024-12-10T16:39:41.514492Z  INFO reth_node_events::node: Executing stage pipeline_stages=4/5 stage=TransactionLookup checkpoint=26479200 target=26479202
2024-12-10T16:39:41.514509Z  INFO sync::stages::transaction_lookup: Updating transaction lookup tx_range=26479200..=26479202
2024-12-10T16:39:41.514566Z  INFO sync::stages::transaction_lookup: Calculating transaction hashes tx_range=1218..1224
2024-12-10T16:39:41.515725Z  INFO sync::stages::transaction_lookup: Inserting hashes append_only=false progress=16.67%
2024-12-10T16:39:41.515750Z  INFO sync::stages::transaction_lookup: Inserting hashes append_only=false progress=33.33%
2024-12-10T16:39:41.515806Z  INFO sync::stages::transaction_lookup: Inserting hashes append_only=false progress=50.00%
2024-12-10T16:39:41.515854Z  INFO sync::stages::transaction_lookup: Inserting hashes append_only=false progress=66.67%
2024-12-10T16:39:41.515869Z  INFO sync::stages::transaction_lookup: Inserting hashes append_only=false progress=83.33%
2024-12-10T16:39:41.516235Z  INFO reth_node_events::node: Finished stage pipeline_stages=4/5 stage=TransactionLookup checkpoint=26479202 target=26479202 stage_progress=100.00%
2024-12-10T16:39:41.702392Z DEBUG storage::db::mdbx: Commit total_duration=186.208958ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 3, audit: 0, write: 0, sync: 12195, ending: 0, whole: 12199, gc_cputime: 130, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 2, work_xtime_cpu: 543, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 88, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-12-10T16:39:41.702614Z  INFO reth_node_events::node: Preparing stage pipeline_stages=5/5 stage=Finish checkpoint=26479200 target=26479202
2024-12-10T16:39:41.702647Z  INFO reth_node_events::node: Executing stage pipeline_stages=5/5 stage=Finish checkpoint=26479200 target=26479202
2024-12-10T16:39:41.702723Z  INFO reth_node_events::node: Finished stage pipeline_stages=5/5 stage=Finish checkpoint=26479202 target=26479202
2024-12-10T16:39:41.888889Z DEBUG storage::db::mdbx: Commit total_duration=186.167333ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 4, audit: 0, write: 0, sync: 12192, ending: 0, whole: 12197, gc_cputime: 181, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 1, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 1, work_rtime_monotonic: 2, work_xtime_cpu: 157, work_rsteps: 1, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 132, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-12-10T16:39:41.889482Z DEBUG downloaders::file: new bytes were read from file max_chunk_byte_len=1000000000 prev_read_bytes_len=5085 new_read_bytes_target_len=0 new_read_bytes_len=0 next_chunk_byte_len=5085 remaining_file_byte_len=0
2024-12-10T16:39:41.889620Z  INFO reth::cli: Importing chain file chunk
2024-12-10T16:39:41.906200Z ERROR reth::cli: Error: file client has no tip

[mattsse]

could you perhaps attach the rlp file?

this looks like a codec issue cc @joshieDo

[debjit-bw]

Hey, here's the RLP file. Note that this file starts from AFTER the last block in reth. Another thing, is it recommended to run it with --no-state?

[debjit-bw]

and this is how i mke this file: https://gist.github.com/debjit-bw/c05fedff2e11fb41e6b17a48fc21214e (original blocks being the ones downloaded using the eth_getBlockByNumber RPC.

[debjit-bw]

hey @mattsse could you look into this?

[debjit-bw]

hey, from what i checked, i can import blocks without transactions. seems to me like the tx encoding is the issue. i'm RLP encoding the signed tx and using that to create the block rlp, so a decoded block would look like this:

 parent   :      a133198478cb01b4585604d07f584633f1f147103b49672d2bd87a5a3ba2c06e 
 uncle    :      1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347 
 coinbase :      0000000000000000000000000000000000000000 
 stat root:      7adc7fb014030c18240e97c5407c07cbf1177b1534c194174f346410234f3596 
 tx root  :      465eb65b1309316e03886fe942b423dab71553ddf02ccf4243e880c9af3274db 
 rcpt root:      f78dfb743fbd92ade140711c8bbc542b5e307f0ab7984eff35d751969fe57efa 
 bloom    :      00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 
 diff     :      0 
 number   :      26478651 
 gas limit:      30000000 
 gas used :      21000 
 timestamp:      1676401685 
 extra    :      Nethermind 
 mix hash :      c5d0c5d500df9ad703ba7f17264fbed0f2f1dd513ef373e508b656ce62beab56 
 nonce    :      0000000000000000 
 base fee/g:     07 
 trxs     :      ["02f870640384908a904084908a904882520894214a33ad7cf5010fc8cee8d501b0ca20c12fdf9e86a11746d62d1080c080a08b911ecfd856becba8508df356629269b5467b8aa7ef6490de4469061817d55fa027c2940c0caef2d43b682e8e6676650facf2150cc749cd79dbec7f134ae11c06"] 
 uncles   :      [] 

i'm not sure what i'm doing wrong, can you pls help me out?

[debjit-bw]

one thing is, for blocks (without txs) even if the import succeeds, in the end it shows:

2024-12-12T18:36:10.893081Z ERROR reth::cli: Chain was partially imported total_decoded_blocks=2 total_imported_blocks=75 total_decoded_txns=0 total_imported_txns=144
2024-12-12T18:36:10.893105Z  INFO reth::cli: Chain file imported total_imported_blocks=75 total_imported_txns=144

Even if the import succeeds.

[debjit-bw]

the issue seems to be the block ordering (blocks should be in reverse order), and different tx types needed to be stored in different ways (for eg RLP-fied for type 1 and 2, only serialised (without RLP-fication) for legacy)

while the ERROR reth::cli: Chain was partially imported issue still persists, it doesn't cause issues, and is fixed with a subsequent reth node

closing this issue