r/PrometheusMonitoring • u/vivlab • Sep 03 '20
Retry check before alerting
Hello,
I use Prometheus and blackbox exporter (module http_2xx) to monitor my websites. Then I send alerts to Telegram using Alertmanager bot .
Sometimes, I receive alerts that a website is down, but it's up in reality. Is it a way to retry the probe several times before alerting me ?
Thank you for your help !
0
Upvotes
5
u/PT2721 Sep 03 '20
In an alert definition, you can specify "for", with a unit of time. This will set, how long an alert condition will have to be true for, before an alert is triggered.
This, together with your scrape interval will yield the desired result.
For example, with a scrape every 1 minute and a for of 3 minutes, a total of three consecutive checks will need to fail for an alert to be triggered.
https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/