You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
It is a well documented problem that a watch connection can miss events in Kubernetes. Due to that, there is an informer pattern that will do a list of watched resources every so often to correct the internal cache.
In Pepr, if the LOG_LEVEL is not set to verbose, it is impossible to understand what is going on with the informer. We need a way to calculate the number of missed events each relist interval so we know if we need to shorten the window.
Metrics we need to see:
cacheMissesPerWindow
retryCounts - If there are many cache misses per window and the retryCount is never reaching the retryLimit then the retryLimit is too high. This is in the case where there is only one event missed and the retryCount keeps getting reset to 0 but events are still being frequently missed. It ultimately means the only way we will reach homeostasis is the relist
Can also tell us to shrink the relist window.
Describe the solution you'd like
Given I am curious about how many events are being missed by the watch connection
When I look at the Pepr metrics
Then I see windows with how many misses occurred during them
Describe alternatives you've considered
(optional) A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
It is a well documented problem that a watch connection can miss events in Kubernetes. Due to that, there is an informer pattern that will do a list of watched resources every so often to correct the internal cache.
In Pepr, if the
LOG_LEVEL
is not set to verbose, it is impossible to understand what is going on with the informer. We need a way to calculate the number of missed events each relist interval so we know if we need to shorten the window.Metrics we need to see:
cacheMissesPerWindow
retryCounts
- If there are many cache misses per window and the retryCount is never reaching the retryLimit then the retryLimit is too high. This is in the case where there is only one event missed and the retryCount keeps getting reset to 0 but events are still being frequently missed. It ultimately means the only way we will reach homeostasis is the relistCan also tell us to shrink the relist window.
Describe the solution you'd like
Describe alternatives you've considered
(optional) A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: