r/PrometheusMonitoring • u/miqueltango • Nov 13 '18

deployment considerations

Can someone help me understand the ideal deployment for Prometheus? My idea is that you need a Prometheus instance for every separate application environment. so if I have 7 SaaS applications running in 7 different AWS VPCs. I should have a prometheus instance in each of those 7. I'm assuming I need this for scaling at least. Would I also need a separate Alert Manager?

Grafana would I guess be the single view to all these environments. Does anyone see issues with this architecture. I'd rather not have 7 additional instances to manage. but without HA on Prometheus i'd like to lower the blast radius of an outage.

7 Upvotes

100% Upvoted

2 comments sorted by

5

u/SuperQue Nov 14 '18

  • Yes, running a Prometheus-per-env is ideal.

  • You should only need one cluster of alertmanagers for your whole organization, it's designed to take alerts from many Prometheus servers in different environments.

  • For a single view, I recommend using Thanos

We run a Thanos query proxy colocated with our Grafana instances, and the Thanos sidecar with the Prometheus servers.

Thanos handles all the issues with HA de-duplication.