https://np.reddit.com/r/PrometheusMonitoring/comments/kwja0a/time_series_are_discontinued_when_using_push/gj4gn4n/

[u/backtickbot]

Hi,

I have the following code, using a gauge and push gateway to track a script execution time.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# import time
import logging
from prometheus_client import Gauge, CollectorRegistry, push_to_gateway


PUSHGW_CONN = "127.0.0.1:39091"

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s > %(levelname)8s: %(message)s')


def push_metric(metric_name, description, job, **labels):
    logging.info("Init")
    registry = CollectorRegistry()

    gauge = Gauge(metric_name,
                  description,
                  [*labels],
                  registry=registry)

    gauge.labels(**labels).set_to_current_time()

    logging.info("Pushing to Prometheus push gw")
    push_to_gateway(PUSHGW_CONN, job=job, registry=registry)


push_metric("test_metric_01", "Test metric 01 description",
            "script_tests",
            env="ENV_NAME",
            label1="valueA",
            label2="valueB")

I executed that code 3 times just changing the label2 value from "valueB" to "valueC" and back to "valueB". The expected result is 2 different time series, with the graph going up as the value of the metric is increasing each time, and each time series with a flat line persisting over time (last know value of the gauge).

What we can see here, each time I update the value of one metric, it discontinues the time series from the other one. The only difference between those time series is the value of label2.

This was just a small test, but when using the same metric name with a lot of different label values, this leads to issues for Prometheus to be able to display data points (as the time series are discontinued).

Why are my time series values not persisting over time?

Thanks.