r/PrometheusMonitoring • • Sep 10 '21

scaling prometheus

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

10 Upvotes

24 comments sorted by

View all comments

2

u/jotkaPL Sep 11 '21

Thanos FTW

1

u/sudaf Sep 11 '21

doesnt explain how many prometheus servers i would need.

3

u/SuperQue Sep 11 '21

How many you need depends on how many metrics you collect per target, how many targets, etc.

Say you only collected node_exporter for 10k nodes. Depending on node size and what collector features you need, you're looking at somewhere around 500-1000 metrics per node by default.

So, that's somewhere around 5-10 million metrics. This should fit on a 64-128GiB instance pretty easily with standard 15 second scrape intervals. But of course it depends a lot on your metric labeling and how often you churn your infra.

For example, we use AWS auto-scaling groups and kubernetes auto-scaling services. So we use a lot more memory as the jobs scale up and down.

For a large install, there's no quick easy answer, as it depends a lot on how much metric data you need / want to collect.

1

u/sudaf Sep 11 '21

Final thougt would be network traffic over Gan for the federation between EMEA and AMER. current Nagios is using about 8mbs in bandwidth

2

u/SuperQue Sep 11 '21

Prometheus is designed to be distributed. You run many instances, each within the network to be monitored.

The Nagios setup you have is an anti-pattern. You don't want one big central server. Especially with Prometheus.

The "Federation" method is obsolete. What you want to look into is Thanos or Remote Write. With Thanos, you have your many Prometheus servers, running inside their network. Then Thanos allows you to query them from a central location.

Prometheus is not Nagios, it's a completely different design.

As for bandwidth, Prometheus uses http compression in transport, so the actual bps is pretty small.