Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust segment num entries dynamically based on bytes. #421

Open
kjnilsson opened this issue Feb 12, 2024 · 0 comments
Open

Adjust segment num entries dynamically based on bytes. #421

kjnilsson opened this issue Feb 12, 2024 · 0 comments

Comments

@kjnilsson
Copy link
Contributor

kjnilsson commented Feb 12, 2024

Is your feature request related to a problem? Please describe.

The segments written by the segment writer process always contain a fixed number of log entries which is configurable on a per ra system basis (segment_max_entries). The number of entries per segment needs to be chosen when a segment is created as the segment file both contains the index and the data.
This is fine if commands written within a system are roughly of the same size and the system is appropriately configured according to the average size.

In RabbitMQ however the size of each enqueue command could vary from a few bytes to several MB which means we could be create both oversized and undersized segments. E.g. a quorum queue with 50MB messages would eventually create a segment of over 200GB in size with a segment_max_entries of 4096. This is clearly not optimal.

Describe the solution you'd like

Each Ra system should have a segment target size that the segment writer will attempt to adjust the number of entries for each segment accordingly to get close to the target size.

To safely do so the starting number of entries for the first segment should be a low value that is doubled or halved based on the final size of the previous segment.

To more accurately size the segment write could even base the size of the results of several past segments should this info be available (as may be the case when the segment writer is writing multiple segments in one "flush")

Describe alternatives you've considered

No response

Additional context

No response

@kjnilsson kjnilsson added this to the 2.10.0 milestone Feb 12, 2024
@michaelklishin michaelklishin removed this from the 2.10.0 milestone May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants