-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add local instance's tags to forwarded metrics #188
base: master
Are you sure you want to change the base?
Conversation
Gerald Rule: Copy Observability on Veneur and Unilog pull requestscc @stripe/observability |
(Oops, accidentally removed this when cleaning up branches.) There are some repercussions to this change. Here's a sample of tags we add to our metrics:
This could cause some sneaky tags to show and create multiple time series where they do not currently exist, some examples:
It seems annoying to filter out some of these but it's not necessarily intended to have this create a few time series from what is currently one aggregated one. (In other words, every time series we aggregate would suddenly become multiple due to AZ being mixed in most host groups.) |
This sounds like it would have a significant impact on the cardinality of some of our already-large metrics, right? Particularly stuff like unilog? |
It would definitely have negative effects. But also sorta feels like a bug I think. As #153 makes clear at least some people expect this behavior. Do we not expect it because we're used to the bug? We could also — and this feels icky — blacklist some tags from global proliferation? In other words, here are my |
So, what we could do is ensure (in our deployment) that tags that are configured on the local host are also set on the global boxes as well (thereby overriding the local ones). That'd probably solve our cardinality concerns. |
Summary
Apply a local veneur's configured tags to metrics that are forwarded.
Motivation
We were asked to do this in #153. It was perhaps an oversight / bug that a local instance's tags are not passed along to the importer.
The idea is that if my local veneur has a tag like
host_type
then the importer will receive it. Today, as far as I can tell, this doesn't happen. We just inherit the tags of the importer at the time it flushes to the sink.Test plan
Added a new unit test, as well as some plumbing that allows future-us to more easily leverage the fixtures HTTP test harness to deal with
/import
calls.r? @aditya-stripe