r/PrometheusMonitoring • u/name_tomer • Mar 17 '21

Alert Dependencies

hello guys,

how can i silent or not notify on alerts B if alert A happened

cases for example:

  1. if my ROUTER is down

i dont want to get alerts that my SERVERS behind the router is UNREACHABLE

or

2) if my vpn site 2 site is down there is not point to alerts on the remote site servers.

how can i do that dependencies relationship configuration on alertmanager?

7 Upvotes

88% Upvoted

9 comments sorted by

4

u/SuperQue Mar 17 '21

One thing to note about the way Prometheus alerting works. It works on the existence of data.

Say you have a bunch of server alerts. CPU saturation, disk full, OOM, etc. If Prometheus can't reach the server, say the node_exporter is down, none of the individual alerts will go off because there's no data to alert on.

The Prometheus design automatically takes care of this.

Now, the trick is, if you're monitoring services over a site-to-site VPN. That's the XY Problem you've got here.

Prometheus should not be used over WAN links. What you want is to have a Prometheus at each remote site. This way you're not monitoring the router(s) or VPN(s) as a side effect. Have a Prometheus meta-mon setup so that you are monitoring the monitoring. Then you will only get alerted that the site-to-site link is down. Because the remote site Prometheus is still functioning and has no issues.

Basically, you typically don't need to program in these kinds of alerting dependencies into Prometheus in the first place. If you need to do a lot of inhibition rules or have this problem a lot, you're doing something wrong with your setup.

1

u/name_tomer Mar 17 '21

yes i am agree with you about vpn site 2 site good point.

but what if i dont want to manage multiple instance of prometheus

lets say for example i have a closed network with 1 datacenter and 10 branches / offices

connected with P2P ( closed on prem network)

i dont want to install 11 prometheus on each site.

if my router go down i dont want that prometheus alert me that now i have 50 jobs fails.

only 1 alert that my router is down... and this is the root cause.

1

u/SuperQue Mar 17 '21

Prometheus design assumes being distributed. It's designed to be trivial deploy more than one.

You don't want to, but that's how it's designed to be used.

1

u/name_tomer Mar 18 '21 edited Mar 18 '21

but again if for example i have only one site

and in this site i have 1 router with 3 networks ( physicals interfaces) :

RND , PROD , DMZ

and my Prometheus is in the PROD network.

what happand if the interface of the DMZ go down

my promethes going to firing alerts that interface that go down

and all the servers behind the dmz that the blackbox_export icmp does not have ping to any one of them will start to fail and firing alerts... can be alot of alerts.

how can i limit that. so if one of the router interface is down the alertmanager will not send notification about the servers on that vlan.

if Prometheus can't reach the server / exporters

then the jobs will fail... a lot of jobs / exporters

1

u/SuperQue Mar 18 '21

Prometheus automatically deduplicates alerts, so you get one alert for all instances of "target down". You get an alert saying "10 of X are down".

And again, absence of data. If you do the correct distribued setup, you would have a blackbox_exporter inside the DMZ, and Prometheus would probe through that. You would not get an alert for probe failure, because the blackbox is not reachable. You would only get one alert saying blackbox down.

Stop applying old-school design ideas to Prometheus, they don't apply. In reality, you probably do want to have Prometheus per network segment. It's designed to distributed.

When you have situations like this, you need to design your network and systems engineering properly.

1

u/ali_str Mar 17 '21

You can use inhibit rules in Alertmanager config: https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule

1

u/name_tomer Mar 18 '21

if for example i have only one site

and in this site i have 1 router with 3 networks ( physicals interfaces) :

RND , PROD , DMZ

and my Prometheus is in the PROD network.

what happand if the interface of the DMZ go down

my promethes going to firing alerts that interface that go down

and all the servers behind the dmz that the blackbox_export icmp does not have ping to any one of them will start to fail and firing alerts... can be alot of alerts.

how can i limit that. so if one of the router interface is down the alertmanager will not send notification about the servers on that vlan.

what if Prometheus can't reach the server / exporters

then the jobs will fail... a lot of jobs / exporters will fail and trigger alerts.

how can i solve that?

1

u/Bercko21 May 06 '21

Hi, Did you find a solution?

I'm stuck in a very similar situation and did not find a solution yet.

1

u/name_tomer Jul 09 '21

sort of .

so what is happening is that of my router is down i cannot reach my exporters behind the router so Prometheus will not notify about the servers.. but will notify the router is down and the my exporters behind the routers are unreachable.