-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf(su): implement SU datastore to run purely on rocksdb #987
Labels
su
ao Scheduler Unit
Comments
VinceJuliano
added a commit
that referenced
this issue
Oct 4, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 4, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 4, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 14, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 15, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 15, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 15, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 15, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 15, 2024
VinceJuliano
added a commit
that referenced
this issue
Oct 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
Under heavy load, the SU database implementation on postgres runs into performance issues. When there are a lot of writes going on the writes become very slow etc... RocksDB was added originally because on the message list query because it contains json data which was stored in postgres, was very slow for large amounts of data.
Solution
To solve this, we can add another RocksDB directory into the SU that will hold the index information. Then we can remove postgres.
implement a new Data Store that runs only on RocksDB. 1 Rocks database for the binary data which already exists for the messages and runs in Blob mode for large data, and another Rocks directory that holds the index information and runs in the regular mode.
Write another binary like the migration binary to migrate the index information. This should be much faster then the one that migrates the Message bundles themselves because it only needs to grab index info which is quicker to fetch from postgres.
The text was updated successfully, but these errors were encountered: