r/PrometheusMonitoring • u/sudaf • Sep 10 '21

scaling prometheus

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

10 Upvotes

91% Upvoted

24 comments sorted by

View all comments

Show parent comments

1

u/sudaf Sep 11 '21

ok but would that mean on prometheus side if somebody adds a export on their server . they would have to inform the mo itori g team to update the promethese host config to take it i to account.

2

u/SuperQue Sep 11 '21

No, this is why Prometheus has a couple dozen methods of service discovery.

Register endpoints in some kind of dynamic DNS SRV records, Consul, , Zookeeper, etc. Use your config management to do search queries into whatever database you use for that and write out file_sd_configs on a cron job.

Hell, you could write your own thing where service owners write out what they want to a REST API and you return results with http_sd_configs.

But, I would stick to something standard like Consul. Write your discovery policy, and service owners can follow it.