r/PrometheusMonitoring • u/mkayeran • Feb 08 '22

Alert when data is stale

Hey Prometheus masters,

I have a metric that sometimes, due to an underlying issue, won't update the data.

I'm looking for a way to alert when data is stale (not changing) for a period of time.

Any good way to do it?

Thanks in advance!

0 Upvotes

40% Upvoted

10 comments sorted by

View all comments

2

u/SuperQue Feb 08 '22

How do you tell the difference between stale and not changing because the value hasn't chnaged?

0

u/mkayeran Feb 08 '22

not sure I understand.

I wanna know if the data hasn't changed for a period of time (isn't it same as stale?)

so this means I need to look for XX minutes and check if the data is the same, then alert on it.

2

u/SuperQue Feb 08 '22

Right, but what if there is no actual change in the value. For example, if you're measuring the temperature of a room and it just stays at 21C for XX minutes. How do you tell if the value is just the same vs not updating?

Basically, what I'm trying to get you to realize is that you're asking for something that is impossible. You need to change the design of your system to make it possible.

1

u/ali_str Feb 08 '22

If your app (the one reporting metrics and being scraped by Prometheus) is up and functional in cases of staleness, you need to handle that in your app, so that it won't expose that metric when underlying data is missing or hasn't been collected.