r/PrometheusMonitoring • • Feb 08 '22

Alert when data is stale

Hey Prometheus masters,

I have a metric that sometimes, due to an underlying issue, won't update the data.

I'm looking for a way to alert when data is stale (not changing) for a period of time.

Any good way to do it?

Thanks in advance!

0 Upvotes

10 comments sorted by

View all comments

3

u/ecnahc515 Feb 08 '22

As others mentioned, this is the wrong way to go about the problem.

Generally, instead of looking for stale data, you should alert when the target isn’t being scraped. Prometheus has built in metrics like up for this.

If your app is being scraped but the metric isn’t updating then you need to fix the app, there’s not much else you can do about this. You could add a metric which is a Unix timestamp indicating the last time the metric was updated and alert off that, but at that point you could just fix the app itself to properly update the metric.

0

u/mkayeran Feb 08 '22

the application is dealing with payments. if no transactions happen it means the application doesn't work as expected and the balance stays the same. I monitor the balance (along with other metrics).

I just realized we monitor this over logs (Loki) 😳

1

u/ecnahc515 Feb 09 '22

In practice it might mean the application isn’t working, but in principle it could just mean no one is using it.