Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Galexie: Optimize resumeable start range based on core's LCL #5518

Open
chowbao opened this issue Nov 5, 2024 · 0 comments
Open

Galexie: Optimize resumeable start range based on core's LCL #5518

chowbao opened this issue Nov 5, 2024 · 0 comments

Comments

@chowbao
Copy link
Contributor

chowbao commented Nov 5, 2024

this feature was discussed here on pr for resumability.

Paraphrasing the requirements from discussion:

if we have a buckets db from the previous run we should query it to find the last closed ledger because it's very likely that the last uploaded ledger is close to the LCL.

We should adjust the start of range to LCL - (bucketSize *2) to narrow down the effective size of range.

Currently, there are ~50 million ledgers in full history. To find the latest ledger in the data lake, it would take log_2(50 * 10^6) which comes out to about 25 requests to GCS. If we use the LCL from captive core, we could potentially reduce the search space to ~ 100s of ledgers or less . log_2(100) comes out to ~ 7 requests to GCS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Backlog
Development

No branches or pull requests

1 participant