Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for query all and scan all #180

Merged
merged 10 commits into from
May 2, 2024

Conversation

danieloh0714
Copy link
Contributor

@danieloh0714 danieloh0714 commented Apr 30, 2024

Add four new functions for getting batched data:

  • queryAll: Executes a query and returns a sequence of pages that contains all results, regardless of page size. New pages will be fetched as needed when the resulting sequence is enumerated.
  • queryAllContents: Executes a query and returns a sequence that contains all results, regardless of page size. New pages will be fetched as needed when the resulting sequence is enumerated.
  • scanAll: Executes a scan and returns a sequence of pages that contains all results, regardless of page size. New pages will be fetched as needed when the resulting sequence is enumerated.
  • scanAllContents: Executes a scan and returns a sequence that contains all results, regardless of page size. New pages will be fetched as needed when the resulting sequence is enumerated.

Copy link
Collaborator

@kyeotic kyeotic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, thank you for putting this together. This is an awesome feature to have.

I could accept this as is, but having built this feature in typescript there is one learning I would like to steal: query limits. The types for this are here, and the implementation is here. These are safety parameters that will cutoff what would otherwise be a complete table scan. This could be done by the caller, but stopping enumeration, but its always nice to be able to specify your guardrails up-front.

That being said, this guardrails can always be added later. If you don't feel like taking on the extra work, just add tests for this and I will take the PR.

Copy link
Collaborator

@kyeotic kyeotic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because scanning can bring back items of the wrong type I had to change the tests. This is not something this PR introduced, its always been a tempest issue.

@kyeotic kyeotic merged commit 0c53a3f into cashapp:main May 2, 2024
2 checks passed
@danieloh0714 danieloh0714 deleted the doh/add-queryAll-scanAll branch May 2, 2024 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants