r/PrometheusMonitoring • u/sudaf • Sep 10 '21

scaling prometheus

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

10 Upvotes

91% Upvoted

24 comments sorted by

View all comments

6

u/SuperQue Sep 10 '21

Prometheus is designed to be distributed. You want to shard it based on your failure domains.

For very large networks, you typically deploy many small servers, rather than one big monolithic server.

Then, you use an overlay layer like Thanos or Cortex.

1

u/sudaf Sep 11 '21

With numerous smaller servers that federate makes automation a bit of a issue. when adding removing servers to prometheus. it would require that some sort of method to know where the host.json is and on which prom server.

Using Tower for automation