Use multi-core to accelerate ledger synchronization

[Jim8y]

Summary or problem description
As is known to the community, it takes a few days to re-sync the neo ledger, it is not only painful, but also dangerous for nodes as they can not get the latest ledger state while re-syncing.

Do you have any solution you want to propose?
In the future we will implement checkpoint for the ledger such that nodes do not need to resync everything. But before that, we can have a much easier solution to speed up the process, which is using the multi-core to synchronize the ledger in parallel.

This idea can be considered as a follow up work of #2919, in which i proposed the transaction execution parallelization, now i think we can push it further to enable blocks to be executed in parallel.

how it works?

• 2*core_number threads process the same number of blocks in parallel
• record the state write
• persist the blocks in order and checks read conflict
• reexecute the blocks that has conflct.

further speed up

We can have a dependency map telling the system which block depends on which blocks to prevent conflict, such that we can better schedule the threads and blocks.

expectations

dozens of times faster to resync.

Neo Version

• Neo 3

Where in the software does this update applies to?

• CLI
• Plugins
• Ledger

[roman-khimov]

1. You can't process block until you have verified its header and this has quite some cost to it, for current Neo blocks it's prevailing executions in fact (try disabling signature verifications in NeoGo to see that).
2. Every block has two execution contexts. In the first one we always burn GAS fees (if there are any transactions). In the second one we always generate some GAS for committee members. Once in a while we generate a new committee. Some oracle (and notary-assisted!) transactions can add activity as well and native contracts can theoretically be updated. There are a lot of dependencies there.
3. IIRC C# node syncs the DB to the underlying storage for every block and it's also very costly and won't go away with your approach.
4. So my expectations are very low, I'd expect more degradation rather than any gains.
5. C# implementation can be 2-3 times faster without this.
6. But you can try, of course, maybe it'll work for some reason.

[cschuchardt88]

We need a new restore blockchain solution from backup. Using the chain.0.acc.zip method is horrible and cost lots of CPU time $$$ for people in the cloud. We need a better system for downloading the blockchain. Something like bittorrent

[roman-khimov]

More like #2373, probably. And we can get it after #1526.

[Jim8y]

More like #2373, probably. And we can get it after #1526.

Another magical time travel~~ an issue in 2021