Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow passing existing dataset into write_fragments and Dataset::commit() #3058

Open
wjones127 opened this issue Oct 28, 2024 · 1 comment · May be fixed by #3119
Open

Allow passing existing dataset into write_fragments and Dataset::commit() #3058

wjones127 opened this issue Oct 28, 2024 · 1 comment · May be fixed by #3119
Assignees
@wjones127
Labels
performance

Comments

@wjones127
Copy link
Contributor

Right now, write_fragments opens the dataset as the first step. This is annoying if you are trying to do a parallel write, since we shouldn't have to repeat those same IOPs for each thread.
@wjones127
Copy link
Contributor Author

There's probably something similar we should do for LanceDataset.commit().

@wjones127 wjones127 changed the title Allow passing existing dataset into write_fragments Allow passing existing dataset into write_fragments and Dataset::commit() Nov 6, 2024
@wjones127 wjones127 self-assigned this Nov 8, 2024
@wjones127 wjones127 linked a pull request Nov 13, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wjones127 wjones127

Labels
performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat!: allow passing down existing dataset for write wjones127/lance
1 participant
@wjones127