Metrics become available only on second .Set()

[marevers]

I am observing some strange behavior using .WithLabelValues and creating GaugeVec metrics using promauto.

Here's a shortened/abbreviated rundown of how I am generating my metrics:

// Define exporter
type exporter struct {
	Config  config
	Reg     *prometheus.Registry
	Metrics struct {
		MetricVec    *prometheus.GaugeVec
	}
}

// Create a registry
func createRegistry() *prometheus.Registry {
	reg := prometheus.NewRegistry()

	reg.MustRegister(
		collectors.NewGoCollector(),
		collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
	)

	return reg
}

// Register a metric with the exporter
func (e *exporter) registerMetrics(labels []string) {
	e.Metrics.MetricVec = promauto.With(e.Reg).NewGaugeVec(prometheus.GaugeOpts{
		Name:      "my_metric",
		Namespace: "my_metric_prefix",
		Help:      "Help of the metric",
	}, labels)
}

// Initialise exporter and register metrics
exporter := &exporter{
	Config: cfg,
	Reg:    createRegistry(),
}

exporter.registerMetrics(labels)

On an interval I am calling the following function (again, abbreviated for readability):

func (e *exporter) calculateMetrics() {
	if someCondition {
		e.Metrics.MetricVec.WithLabelValues(labelValues...).Set(1)
	} else {
		e.Metrics.MetricVec.WithLabelValues(labelValues...).Set(0)
	}
}

Now, interestingly enough, only on the second time calculateMetrics is called, my metrics become available/visible (on promhttp.HandlerFor(exporter.Reg, promhttp.HandlerOpts{}). The very first time, the metrics are seemingly correctly set but they are not visible. This causes the exporter to be delayed in providing the metrics, e.g. if the interval is set to 5 minutes, the metrics become available after 10 minutes.

Is this a usage error, or is the behavior somehow bugged?

[dosubot]

Hey @marevers! I'm here to help you with any bugs, questions, or contributions. Let's tackle this together!

This behavior might be related to how metrics are initialized and registered in Prometheus when using GaugeVec with WithLabelValues. A similar issue was discussed in a closed issue on the client_golang repository, where metrics like CounterVec were not included in the Gather results unless WithLabelValues was called at least once ^[1].

In your case, it seems that the metrics are not being initialized or registered in the registry until the second call to Set. This could be due to the fact that the metric is only being created and registered in the registry when WithLabelValues is called for the first time, but it might not be fully visible until the second call.

To address this, you can try explicitly initializing the metric with a default value (e.g., 0) right after registering it. This ensures that the metric is present in the registry from the start. Here's a possible modification to your code:

// Register a metric with the exporter
func (e *exporter) registerMetrics(labels []string) {
	e.Metrics.MetricVec = promauto.With(e.Reg).NewGaugeVec(prometheus.GaugeOpts{
		Name:      "my_metric",
		Namespace: "my_metric_prefix",
		Help:      "Help of the metric",
	}, labels)

	// Initialize the metric with default label values
	e.Metrics.MetricVec.WithLabelValues(make([]string, len(labels))...).Set(0)
}

This approach initializes the metric with default label values, ensuring it is visible in the registry from the beginning.

To continue talking to Dosu, mention @dosu.

[marevers]

This solution unfortunately doesn't work, because e.Metrics.MetricVec.WithLabelValues(make([]string, len(labels))...).Set(0) applies the labels with empty label values. As the correct label values are not yet defined - they are retrieved in the course of calculateMetrics() - it still results in the metrics appearing on the second run of the aforementioned function.

I have applied a provisional fix by just running calculateMetrics() twice before starting the interval goroutine, but I do wonder if there is a proper, idiomatic way of handling this.

[bwplotka]

Thanks for reporting!

e.Metrics.MetricVec = promauto.With(e.Reg).NewGaugeVec(prometheus.GaugeOpts{
		Name:      "my_metric",
		Namespace: "my_metric_prefix",
		Help:      "Help of the metric",
	}, labels)

So just this line obv will not give you the my_metric{<your dynamic label>, hope that's clear. The first one calculateMetrics (or WithLabelValues(labelValues...) with or without set) should give you those metrics.

I wonder, maybe there's a bug in your calculateMetrics() with the labelValues you are passing? But even then, even empty string should appear.

We can create a quick unit test to verify here (help wanted) and try to repro this in isolation, but you are right, first WithLabelValues(labelValues...) (with or without Set) should immediately create a gauge.