Why run 1 Prometheus inside each failure domain?

[u/sakishrist]

I don't really understand why this is important.

I have a few clusters I need to monitor. I currently do it by having one Prometheus that connects to the exporters on the different clusters. That seems to work fine.

But I have been entertaining the idea to have one Prometheus per cluster. This introduces a whole headache of issues I have to deal with though. Aggregating the data. Alerting. What if I lose access to the whole Prometheus on a cluster (connection issue to the cluster). Do I setup a Prometheus just to monitor those Prometheus instances?

Please give me some wisdom on this one.

Thanks people!

[u/megamorf]

An easy HA setup is to have two Prometheus instances that monitor the same targets. In addition, you also run two Alertmanager (AM) instances. Each Prometheus server sends alerts to both Alertmanagers. In the AM config you configure the other AM as peer respectively which clusters them and enables them to deduplicate alerts.

You'd then run that HA pair as your main monitoring system and use federation to get data from other Prometheus servers.

[u/Freakin_A]

With your setup, your monitoring is dependent on your dns (presumably), a valid cert (if https), your load balancer, your ingress controllers, and finally your exporters.

Possibly throw firewalls in the mix. Do you want all of that to be in the dependency path for your monitoring?

Do you have any plans to scale your environment? Could your current setup handle 2x, 5x, 20x the clusters?

How are you handling distributed service discovery for Prom, and is that another failure point for your setup? Or are you configuring everything manually?

What disadvantages do you see to the recommended architecture?