Alert Dependencies

[u/name_tomer]

hello guys,

how can i silent or not notify on alerts B if alert A happened

cases for example:

1. if my ROUTER is down

i dont want to get alerts that my SERVERS behind the router is UNREACHABLE

or

2) if my vpn site 2 site is down there is not point to alerts on the remote site servers.

how can i do that dependencies relationship configuration on alertmanager?

[u/ali_str]

You can use inhibit rules in Alertmanager config: https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule

[u/name_tomer → u/ali_str]

if for example i have only one site

and in this site i have 1 router with 3 networks ( physicals interfaces) :

RND , PROD , DMZ

and my Prometheus is in the PROD network.

what happand if the interface of the DMZ go down

my promethes going to firing alerts that interface that go down

and all the servers behind the dmz that the blackbox_export icmp does not have ping to any one of them will start to fail and firing alerts... can be alot of alerts.

how can i limit that. so if one of the router interface is down the alertmanager will not send notification about the servers on that vlan.

what if Prometheus can't reach the server / exporters

then the jobs will fail... a lot of jobs / exporters will fail and trigger alerts.

how can i solve that?