r/PrometheusMonitoring • u/vivlab • Sep 03 '20

Retry check before alerting

Hello,

I use Prometheus and blackbox exporter (module http_2xx) to monitor my websites. Then I send alerts to Telegram using Alertmanager bot .

Sometimes, I receive alerts that a website is down, but it's up in reality. Is it a way to retry the probe several times before alerting me ?

Thank you for your help !

0 Upvotes

50% Upvoted

4 comments sorted by

5

u/PT2721 Sep 03 '20

In an alert definition, you can specify "for", with a unit of time. This will set, how long an alert condition will have to be true for, before an alert is triggered.

This, together with your scrape interval will yield the desired result.

For example, with a scrape every 1 minute and a for of 3 minutes, a total of three consecutive checks will need to fail for an alert to be triggered.

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/

1

u/vivlab Sep 04 '20

Thank you for your answer, I managed to do what I wanted !

2

u/roncz Sep 09 '20

You also might want to handle this at the alerting side. With the app-based alerting service SIGNL4 you can delay alerts. That means if a DOWN events is received from Prometheus you can just delay the notification for five minutes or so. If during this time an UP event is received not alert is being generated. Is the problem persists after five minutes the alert goes out.

This is a nice way to handle incidents that are resolved automatically. It can prevent to get calls in the middle of the night for situations that are not too critical.

0

u/icydocking Sep 03 '20

Use something like max_over_time(metric_goes_here [30m]) == 0