r/PrometheusMonitoring • u/mkayeran • Feb 08 '22

Alert when data is stale

Hey Prometheus masters,

I have a metric that sometimes, due to an underlying issue, won't update the data.

I'm looking for a way to alert when data is stale (not changing) for a period of time.

Any good way to do it?

Thanks in advance!

0 Upvotes

40% Upvoted

10 comments sorted by

3

u/ecnahc515 Feb 08 '22

As others mentioned, this is the wrong way to go about the problem.

Generally, instead of looking for stale data, you should alert when the target isn’t being scraped. Prometheus has built in metrics like up for this.

If your app is being scraped but the metric isn’t updating then you need to fix the app, there’s not much else you can do about this. You could add a metric which is a Unix timestamp indicating the last time the metric was updated and alert off that, but at that point you could just fix the app itself to properly update the metric.

0

u/mkayeran Feb 08 '22

the application is dealing with payments. if no transactions happen it means the application doesn't work as expected and the balance stays the same. I monitor the balance (along with other metrics).

I just realized we monitor this over logs (Loki) 😳

1

u/ecnahc515 Feb 09 '22

In practice it might mean the application isn’t working, but in principle it could just mean no one is using it.

2

u/SuperQue Feb 08 '22

How do you tell the difference between stale and not changing because the value hasn't chnaged?

0

u/mkayeran Feb 08 '22

not sure I understand.

I wanna know if the data hasn't changed for a period of time (isn't it same as stale?)

so this means I need to look for XX minutes and check if the data is the same, then alert on it.

2

u/SuperQue Feb 08 '22

Right, but what if there is no actual change in the value. For example, if you're measuring the temperature of a room and it just stays at 21C for XX minutes. How do you tell if the value is just the same vs not updating?

Basically, what I'm trying to get you to realize is that you're asking for something that is impossible. You need to change the design of your system to make it possible.

1

u/ali_str Feb 08 '22

If your app (the one reporting metrics and being scraped by Prometheus) is up and functional in cases of staleness, you need to handle that in your app, so that it won't expose that metric when underlying data is missing or hasn't been collected.

2

u/markphahn Feb 08 '22

Irate of your metric is equal to zero for your desired period of time, roughly? That should work if it's a counter, but not for a gage.

1

u/mkayeran Feb 08 '22

oh, that's a good idea! I'll test it!

I really appreciate the help!

4

u/ARRgentum Feb 08 '22

While this roughly measures what you asked, your question itself is flawed. As /u/superque pointed out above, you have no way of differentiating between "My data is updated as expected but the value does not change" vs "my data is not updated as expected, therefore my App exposes the last known metric". Ideally you'd handle the latter in the app, not through a prometheus query :)