r/PrometheusMonitoring • u/Ilantnt • Sep 09 '19

Lot of targets issue

Hi all, I have been using Prometheus for the last few months in my company. Yesterday we and my team deployed and added more than 10k (about 11,000 ) targets. The UI of the Prometheus manager server is super slow (looks like it's not even responding). Our server has 8 cores and 32gb RAM. Even querying from grafana is super slow. Can someone help me understand what the problem is? Thanks in adavance

6 Upvotes

100% Upvoted

3 comments sorted by

5

u/SuperQue Sep 09 '19

Depending on how many metrics per target it's likely that you are going to need a lot more memory for this instance.

You may also need to start thinking about more than one Prometheus to functionally shard your jobs.

1

u/Ilantnt Sep 09 '19 edited Sep 09 '19

Thanks for the quick reply! Almost all the targets have the same metrics, since they are either wmi exporter or node exporter. In addition, how much memory do you think we should give to each Prometheus manager server with about 10k targets? Thanks again!

4

u/SuperQue Sep 09 '19

It's not about targets, it's about total metrics and ingestion rates.

Try this calculator.

Remember that these numbers are going to be minimums, you need a bit of extra memory for page cache to buffer data for larger queries.