Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

saveAll should use Mongo's insertMany method for bulk saving new documents #9143

Open
pocketcolin opened this issue Jun 3, 2024 · 3 comments
Labels
type:feature New feature or improvement of existing feature

Comments

@pocketcolin
Copy link

New Feature / Enhancement Checklist

Current Limitation

Unless I'm misreading the code, it appears that under the hood the Parse.Object.saveAll method is making many synchronous requests instead of batching them together into a single atomic Bulk request. I see in the MongoStorageAdapter that Parse uses updateMany and even deleteMany, but I don't see insertMany anywhere and the only create function appears to be createObject (singular). The batch endpoint handler also appears to be running the requests separately (here). Please let me know if I'm missing something that we are actually inserting documents in bulk!

Feature / Enhancement Description

By leveraging Mongo's native insert bulk function, we'd reduce number of connections and speed up large requests.

Example Use Case

I need to insert 1000 new documents but want the inserts to happen as close to simultaneously as possible.

Alternatives / Workarounds

Using the Mongo driver directly.

Copy link

Thanks for opening this issue!

  • 🎉 We are excited about your ideas for improvement!

@mtrezza
Copy link
Member

mtrezza commented Jun 4, 2024

Haven't looked into this now, but one reason may be what the MongoDB driver returns in its response. Depending on that this may need to be implemented as a new separate Parse method, or can be implemented as an improvement of the existing method.

@mtrezza mtrezza added the type:feature New feature or improvement of existing feature label Jun 4, 2024
@pocketcolin
Copy link
Author

Thanks for the insight, @mtrezza . That definitely makes sense. I started digging into the functionality because we noticed that when running a saveAll on a large array of new documents the new documents would appear in Mongo a few at a time as opposed to all at once. Mongo does some batching under the hood and indexes can take a second to update when large amounts of data is inserted at once, but the tests seem to indicate that at least some of the delay is on Parse's end. That said, when I really dug into the Mongo driver in Parse, I definitely got lost trying to follow the batch paths so it's totally possible I missed something and Parse is batching those requests. If you have time to verify one way or another, I might be able to help out with this improvement (maybe a new optional config param on saveAll?)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:feature New feature or improvement of existing feature
Projects
None yet
Development

No branches or pull requests

2 participants