Allowing user-defined labels with runtime cardinality limits

[emschwartz]

@keturiosakys and I just had a conversation with someone who was suggesting allowing users to add custom labels to metrics. Cardinality is obviously the main issue, and we'd like to avoid letting people easily shoot themselves in the foot and blowing their Prometheus instances.

One potential approach for this would be to allow users to pass in custom labels but have a pre-defined cardinality limit on that label. Before adding a new in-memory counter for a different variant of the label, we would check if we have too many of them already. If so, we would probably just not set the label (or maybe have a default behavior like having an "other" variant).

There's obviously some balance between wanting to not throw out data the user has said they want to collect, while also protecting their Prometheus instance because it's better to keep the instance alive and be missing a little bit of data rather than bring the whole thing down.

One interesting realization about this is that the limit can only be applied on a per-instance basis, so it wouldn't protect you if you have a lot of different instances and each one has different sets of label variants that it's tracking. However, this is actually fine. Because Prometheus adds instance-specific labels, it's already storing all of the data from each instance as separate time series. Therefore, limiting the cardinality of the series on a per-instance basis using limits stored and tracked in-memory would be fine.

This idea almost seems too obvious, I wonder why Prometheus clients don't do anything along these lines already. Maybe there's some big flaw in this line of thinking... 🤔