Cooperative stickies w/ metrics #43
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
omnibs
force-pushed
the
qox-cooperative-stickies
branch
from
cd89e41 to
4455a57
Compare
stoeffel
reviewed
Apr 26, 2022
| |> map (\_ -> ()) | ||
| flushProducer :: Producer.KafkaProducer -> Prelude.IO () | ||
| flushProducer producer = do | ||
| Producer.flushProducer producer |
There was a problem hiding this comment.
We've used flushProducer before in order to use sync sending messages. I think it's also a blocking function and I think to remember that it also slowed down sending immensely. ℹ️
We're gettingt this error: All partition.assignment.strategy (range,cooperative-sticky) assignors must have the same protocol type, online migration between assignors with different protocol types is not supported Some minimal reading of rdkafka made us believe range is EAGER and cooperative-sticky is .. well .. COOPERATIVE, which are different protocol types. This means we can't use both at the same time and in practice, means we need downtime on workers when deploying this change. Not terrible tbh.
Extra props doesn't set things up correctly to use incremental_assign and friends.
We've switched to the only remaining pollEvent proxy: flushProducer It might be fine to flush 'em every 100ms. We'll see.
stoeffel
force-pushed
the
qox-cooperative-stickies
branch
from
0aff8f2 to
da82ac6
Compare
omnibs
added
the
Hold
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Try out our fork of hw-kafka allowing cooperative rebalance strategy, based off of our WIP PR exposing rdkafka metrics, so we can understand what's up with rdkafka under the hood.