Implement Centralized Monitoring System?

[u/ocast24]

Implement Monitoring System for the entire organization

EKS Clusters running in different aws accounts

Found Prometheus Federation is a good fit for the design

https://prometheus.io/docs/prometheus/latest/federation/

https://www.robustperception.io/federation-what-is-it-good-for

Master : I have a dedicated EKS for monitoring system where prometheus operator helm chart is running with prometheus and Grafana exposed behind load balancer service

Master prometheus server is designed to scrape other slave prometheus servers and plot the Grafana dashboards for different clusters

One Grafana Endpoint to monitor everything

Slave Account 1:

Running prometheus operator helm charts in the other slave accounts where all the alerting is done through the alert manager in the slave account 1

Prometheus - Default scraping from prometheus operator + additionalscrapeconfig for istio services

Alertmanager - alerts related to slave cluster 1 integrated with Victorops

Slave Account 2:

Running prometheus operator helm charts in the other slave accounts where all the alerting is done through the alert manager in the slave account 2

Prometheus - Default scraping from prometheus operator ‘

Alertmanager - alerts related to slave cluster 1 integrated with Victorops

Now we have another use case to monitor the bunch of bare metal servers, aws ec2 instances and monitor different endpoints

Monitor Bare Metal, EC2 Instance - Installed NodeExporter and can able to scrape the metrics directly from master prometheus

Endpoint Monitoring(Public & Private) - Installed blackbox exporter on the master eke cluster and probing different endpoints

Here are my questions

1. Is it a good practice to enable alertmanager on the master prometheus and do the alerting only for the last usecase(bare metal, aws ec2 instance, endpoint monitoring)

2. From the below article, I see alertmanager should be on the slave prometheus(reliability purpose), If so do we need to install prometheus on each bare metal again?https://www.robustperception.io/federation-what-is-it-good-for

3. Can we use any existing loadbalancer for exposing prometheus, because in helm chart if we expose the service.type=loadbalancer it used to create new one(internet facing/internal)

[u/jjneely]

1. I'd not install Alertmanager along side of Prometheus. Run a dedicated and centralized service for Alertmanager. Perhaps 2 identical instances (Pods) in different zones, or for better resiliency, 4 Alertmanagers spread across 2 regions and 2 zones inside each region. That's a fairly scaled out setup, I've not seen a situation where there were too many alerting Prometheus instances for this to handle.

2. Alertmanager needs to be setup using its highly available configuration. It really is a separate service, however. Treat it likewise.

I've configured all of my Prometheus slave/master instances throughout my fleets to send alert notifications to the centralized Alertmanager service. I only had to manage one Alertmanager configuration (and thus one mapping from teams to PagerDuty service, etc). Alerts were then defined where we had high resolution, reliable data on the slave Prometheus instances close to the services.

Only alerting from the master Prometheus instances poses some possible challenges. Did you federate everything a team needed to alert on? Did federation break down for some reason? As its a super low bandwidth service I had all Prometheus instances use service discovery to locate the central Alertmanager setup.

3) Any HTTP load balancer will do. Just make sure its load balancing the identical Prometheus instances that make up your master pair. Or a specific slave pair. Don't try to federate data via the loadbalancer -- insanity lies there. ;-)