Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services/galexie: Simplify resumability logic #5547

Open
urvisavla opened this issue Dec 10, 2024 · 0 comments
Open

services/galexie: Simplify resumability logic #5547

urvisavla opened this issue Dec 10, 2024 · 0 comments

Comments

@urvisavla
Copy link
Contributor

What Problem Does Your Feature Solve?

The current resumability logic uses binary search to find the first missing ledger in a given range. While this approach is very efficient and independent of the storage backend, it can be unintuitive for users. It also requires that there are no multiple gaps within the range being exported meets the following precondition:
This feature requires ledgers to be present on the remote data store for some (possibly empty) prefix of the requested range and then absent for the (possibly empty) remainder.

What Would You Like to See?

Reimplement resumability by replacing binary search with a method that lists ledger files within the given range to identify the most recent ledger. The object naming convention we have chosen will ensure that the objects can be lexicographically sorted. This approach is more intuitive as well as deterministic. For bounded mode, the new logic would be efficient since we can list all objects in sorted order with the latest ledger first. However, for unbounded mode, we’ll need to find the most recent ledger that is less than or equal to the end ledger which requires further consideration.

We can also use a hybrid approach, using the listing method for storage backends like GCS that support sorted listing and defaulting to binary search for backends that do not.

What Alternatives Are There?

keep the current binary search implementation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant