Though kafka i am publishing 1lakh events for same bucket, but only 50K jobs are getting stored

[abhisheksahani]

Though kafka i am publishing 1lakh events for same bucket, but only 50K jobs are getting stored

[sandeepmalik]

Hi Abhishek,

Are you publishing events scheduled in very near future? Most likely by the time the remaining 50K messages are read from kafka they might have expired and in that BigBen just triggers them without scheduling as there's no time to schedule.

The approach to just trigger lapsed events rather than storing them first was to provide maximum drainage capability otherwise BigBen might keep on trying to catchup but looks like this is causing confusion. I'm working on adding support for storing the lapsed events as well.