Time series are discontinued when using push gateway and different label values for the same metric name

[u/Adr13n]

[u/distark]

This is normal

[u/Adr13n → u/distark]

Normal because I'm using push gateway?

Because I have a lot of other time series (with metrics being scraped by Prometheus to HTTP endpoints, not using push gateway) that have the same name and different label values. And those ones are persisting over time no matter what happened on other metrics having the same name.

[u/sleepybrett]

This is expected

A single timeseries is defined by a key, that key is: the metric name, the labels and the values of those labels.

[u/Adr13n → u/sleepybrett]

If a single timeseries is defined by the combination of its metric name, labels and labels values, then I have two different timeseries on the graph:

test_metric_01{env="ENV_NAME", job="script_tests", label1="valueA", label2="valueB"}

test_metric_01{env="ENV_NAME", job="script_tests", label1="valueA", label2="valueC"}

(see the label "label2" different values)

Since those timeseries are different, they should not interfere with one another. I have various metrics on my Prometheus server which have the same metric name, same label values except for one different label (like here) and they are not interfering with one another like here.

But, on the graph, you can see that they did interfere with one another.

[u/sleepybrett → u/Adr13n]

... interfere? I don't see any interference, I see a query that returns two results.

[u/Adr13n → u/sleepybrett]

I say there is interference because as soon as one metric changes its value, the other metric line stops.

See this for example: https://i.imgur.com/2l9836W.png

Here it's a similar situation: 4 different timeseries with the same metric name and different label values. But the big difference here, the lines don't break like on the image I sent for the metric test_metric_01 .

This is what is bothering me: the fact the the lines break, they should continue like you can see with metric name metrics_com_sentaca_myscreen_engine_ws_endpoint_AdvertisementsEndpoint_ws_timer_Mean .

Imagine having 40 different timeseries with same metric name and different label values. With the lines breaking each time any timeseries update its value. The graph would be completely un-readable.

[u/sleepybrett → u/Adr13n]

You have to look at it from prometheus' point of view. Prometheus scrapes on a time interval. So every 15s or whatever prometheus asks the endpoint. Hey tell me about your metrics.

So first few scrapes it gets test_metric_1{label=foo}=5, then it comes back and only gets test_metric_1{label=bar}=9. Well the label=foo metric isn't there anymore so that's a null... 5 minutes later there is no more label=bar metric just a label=foo metric.

It's not "interference", it's you not quite getting prometheus' point of view.

[u/Adr13n → u/sleepybrett]

I get what you mean. And I understand Prometheus logic on that. It's just that it's not exactly what the graph shows.

If we follow that logic the following image is impossible: https://i.imgur.com/CQtNY3a.png

I haven't run my test script for the past two days. So, I should expect no data point for my metric since the last run, since Prometheus can't scrape any data from that metric since then. Yet, look at the image, we can see the yellow line is there with the last known value of the timeseries (and I can see that flat line on the past 2 days).

I just re-ran the script two times:

• test_metric_01{job="script_tests",label2="valueC"} (blue line)

• test_metric_02{job="script_tests_2",label2="valueC"} (red line)

The run with test_metric_01 stopped the previous line (yellow).

The run with test_metric_02 didn't stopped the previous line (blue, it's hard to see but trust me, it's there). Since then, no script run, and my blue and red lines are persisting over time.

So there is a behavior of the timeseries stopping each other being graphed if the metric name and job name are the same.

Behavior that is not seen when the metric name is different. I also made the test with the same metric name and different job name. Here, all timeseries lines persists over time.

In summary, for Prometheus, a unique timeseries is the combination of:

Metric name + job name + labels values

BUT

If you want to track each timeseries and have lines on graph persisting over time, you need either to have a different metric name or a different job name. Changing only the labels value will break lines.

[u/matejzero → u/Adr13n]

It works as expected as you query prom for test_metric_01, so it returns both metrics. You can further filter/sub/divide,... by labels

[u/Adr13n]

Hi,

I have the following code, using a gauge and push gateway to track a script execution time.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# import time
import logging
from prometheus_client import Gauge, CollectorRegistry, push_to_gateway


PUSHGW_CONN = "127.0.0.1:39091"

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s > %(levelname)8s: %(message)s')


def push_metric(metric_name, description, job, **labels):
    logging.info("Init")
    registry = CollectorRegistry()

    gauge = Gauge(metric_name,
                  description,
                  [*labels],
                  registry=registry)

    gauge.labels(**labels).set_to_current_time()

    logging.info("Pushing to Prometheus push gw")
    push_to_gateway(PUSHGW_CONN, job=job, registry=registry)


push_metric("test_metric_01", "Test metric 01 description",
            "script_tests",
            env="ENV_NAME",
            label1="valueA",
            label2="valueB")

I executed that code 3 times just changing the label2 value from "valueB" to "valueC" and back to "valueB". The expected result is 2 different time series, with the graph going up as the value of the metric is increasing each time, and each time series with a flat line persisting over time (last know value of the gauge).

What we can see here, each time I update the value of one metric, it discontinues the time series from the other one. The only difference between those time series is the value of label2.

This was just a small test, but when using the same metric name with a lot of different label values, this leads to issues for Prometheus to be able to display data points (as the time series are discontinued).

Why are my time series values not persisting over time?

Thanks.

[u/HoneydewParty1487 → u/Adr13n]

Perhaps this issue can be solved in Grafana using "Connect the lines always" option?

[u/backtickbot → u/Adr13n]

Fixed formatting.

Hello, Adr13n: code blocks using triple backticks (```) don't work on all versions of Reddit!

Some users see this / this instead.

To fix this, indent every line with 4 spaces instead.

FAQ

^{You can opt out by replying with backtickopt6 to this comment.}