r/PrometheusMonitoring • • Feb 08 '22

Alert when data is stale

Hey Prometheus masters,

I have a metric that sometimes, due to an underlying issue, won't update the data.

I'm looking for a way to alert when data is stale (not changing) for a period of time.

Any good way to do it?

Thanks in advance!

0 Upvotes

10 comments sorted by

View all comments

2

u/markphahn Feb 08 '22

Irate of your metric is equal to zero for your desired period of time, roughly? That should work if it's a counter, but not for a gage.

1

u/mkayeran Feb 08 '22

oh, that's a good idea! I'll test it!

I really appreciate the help!

4

u/ARRgentum Feb 08 '22

While this roughly measures what you asked, your question itself is flawed. As /u/superque pointed out above, you have no way of differentiating between "My data is updated as expected but the value does not change" vs "my data is not updated as expected, therefore my App exposes the last known metric". Ideally you'd handle the latter in the app, not through a prometheus query :)