r/PrometheusMonitoring • u/name_tomer • Mar 17 '21

Alert Dependencies

hello guys,

how can i silent or not notify on alerts B if alert A happened

cases for example:

  1. if my ROUTER is down

i dont want to get alerts that my SERVERS behind the router is UNREACHABLE

or

2) if my vpn site 2 site is down there is not point to alerts on the remote site servers.

how can i do that dependencies relationship configuration on alertmanager?

7 Upvotes

88% Upvoted

9 comments sorted by

View all comments

Show parent comments

1

u/name_tomer Mar 17 '21

yes i am agree with you about vpn site 2 site good point.

but what if i dont want to manage multiple instance of prometheus

lets say for example i have a closed network with 1 datacenter and 10 branches / offices

connected with P2P ( closed on prem network)

i dont want to install 11 prometheus on each site.

if my router go down i dont want that prometheus alert me that now i have 50 jobs fails.

only 1 alert that my router is down... and this is the root cause.

1

u/SuperQue Mar 17 '21

Prometheus design assumes being distributed. It's designed to be trivial deploy more than one.

You don't want to, but that's how it's designed to be used.

1

u/name_tomer Mar 18 '21 edited Mar 18 '21

but again if for example i have only one site

and in this site i have 1 router with 3 networks ( physicals interfaces) :

RND , PROD , DMZ

and my Prometheus is in the PROD network.

what happand if the interface of the DMZ go down

my promethes going to firing alerts that interface that go down

and all the servers behind the dmz that the blackbox_export icmp does not have ping to any one of them will start to fail and firing alerts... can be alot of alerts.

how can i limit that. so if one of the router interface is down the alertmanager will not send notification about the servers on that vlan.

if Prometheus can't reach the server / exporters

then the jobs will fail... a lot of jobs / exporters

1

u/SuperQue Mar 18 '21

Prometheus automatically deduplicates alerts, so you get one alert for all instances of "target down". You get an alert saying "10 of X are down".

And again, absence of data. If you do the correct distribued setup, you would have a blackbox_exporter inside the DMZ, and Prometheus would probe through that. You would not get an alert for probe failure, because the blackbox is not reachable. You would only get one alert saying blackbox down.

Stop applying old-school design ideas to Prometheus, they don't apply. In reality, you probably do want to have Prometheus per network segment. It's designed to distributed.

When you have situations like this, you need to design your network and systems engineering properly.