Create prometheus metrics for missed informer events during a relist interval

[cmwylie19]

Is your feature request related to a problem? Please describe.

It is a well documented problem that a watch connection can miss events in Kubernetes. Due to that, there is an informer pattern that will do a list of watched resources every so often to correct the internal cache.

In Pepr, if the LOG_LEVEL is not set to verbose, it is impossible to understand what is going on with the informer. We need a way to calculate the number of missed events each relist interval so we know if we need to shorten the window.

Metrics we need to see:

• cacheMissesPerWindow
• retryCounts - If there are many cache misses per window and the retryCount is never reaching the retryLimit then the retryLimit is too high. This is in the case where there is only one event missed and the retryCount keeps getting reset to 0 but events are still being frequently missed. It ultimately means the only way we will reach homeostasis is the relist

Can also tell us to shrink the relist window.

Describe the solution you'd like

• Given I am curious about how many events are being missed by the watch connection
• When I look at the Pepr metrics
• Then I see windows with how many misses occurred during them

Describe alternatives you've considered

(optional) A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.