How we can control message rate for input/output topics

[ppilev]

hello,

I've read the docs about the Threading model but couldn't figure out how we can control the message rate of input/output topics.

I'm doing left-join for stream A which might have a new record every 1-2 seconds with another stream B receiving about 40 records per second (one record every 25 ms)
When tested with slower rates I've noticed that it's certainly that I'll can see "left record data - <NULL>" in the output topic, however with higher message rates it seems the records are somehow "aggregated" behind the scene and I in the output topic I got directly "left record data - right record data" .

I had similar "problem" when experimenting with KSQL and found I need to modify these two settings in order to achieve what I needed:

SET 'commit.interval.ms'='10';
SET 'ksql.streams.cache.max.bytes.buffering'='0';

Is there anything like that in Streamiz I can use?

Thanks!

[LGouellec]

Hi @ppilev ,

For now Streamiz doesn't support caching. It's plan for 1.5.0 release.

Maybe you can consider use MaxTaskIdleMs : https://lgouellec.github.io/kafka-streams-dotnet/stream-configuration.html#maxtaskidlems

https://medium.com/airwallex-engineering/kafka-its-about-time-part-3-6b3163618798

Best regards,

[ppilev]

hi @LGouellec,

that definitely changed the behavior when running against single Kafka broker running locally in Docker, however I still cannot see "left record data - <NULL>" records when run my app against AWS MSK cluster (3 brokers in 3 availability zones and ReplicationFactor = 3)

is there some other setting which potentially can help achieving what I expect when running against aws msk cluster?

regards,

[LGouellec]

Hi @ppilev ,

I don't understand your problem. What do you want to do ? If the join left-right match, it can be great no ?

The consumer fetching from your broker is not deterministic, meaning if you start X times your application with the same both input topics, you can have multiple result in the output (same for KStreams java, Ksqldb).

Best regards,

[ppilev]

yeah, perhaps it's a timing thing and since it's expected to behave in a non-deterministic way then we just have to accept it as it is.
thank you for your time

best regards,