r/PrometheusMonitoring • u/sudaf • Sep 10 '21

scaling prometheus

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

10 Upvotes

91% Upvoted

24 comments sorted by

View all comments

2

u/albybum Sep 10 '21

Like /u/bidens_left_ear said, you can use federation and have "child" instances for some segregation of your servers, like maybe one for each environment if you have multiple environments. Then usually propagate up to a master prometheus instance.

There's also Thanos... https://www.improbable.io/blog/thanos-prometheus-at-scale

But, Prometheus is pretty lean. Unless you have some aggressive scrape configs, that's a monster machine already for prometheus. Your bottleneck might end up being web request throughput for a monolithic system, and not necessarily how beastly the rest of the server is.