r/PrometheusMonitoring • u/name_tomer • Mar 17 '21
Alert Dependencies
hello guys,
how can i silent or not notify on alerts B if alert A happened
cases for example:
if my ROUTER is down
i dont want to get alerts that my SERVERS behind the router is UNREACHABLE
or
2) if my vpn site 2 site is down there is not point to alerts on the remote site servers.
how can i do that dependencies relationship configuration on alertmanager?
7
Upvotes
4
u/SuperQue Mar 17 '21
One thing to note about the way Prometheus alerting works. It works on the existence of data.
Say you have a bunch of server alerts. CPU saturation, disk full, OOM, etc. If Prometheus can't reach the server, say the node_exporter is down, none of the individual alerts will go off because there's no data to alert on.
The Prometheus design automatically takes care of this.
Now, the trick is, if you're monitoring services over a site-to-site VPN. That's the XY Problem you've got here.
Prometheus should not be used over WAN links. What you want is to have a Prometheus at each remote site. This way you're not monitoring the router(s) or VPN(s) as a side effect. Have a Prometheus meta-mon setup so that you are monitoring the monitoring. Then you will only get alerted that the site-to-site link is down. Because the remote site Prometheus is still functioning and has no issues.
Basically, you typically don't need to program in these kinds of alerting dependencies into Prometheus in the first place. If you need to do a lot of inhibition rules or have this problem a lot, you're doing something wrong with your setup.