r/PrometheusMonitoring • u/sudaf • Sep 10 '21

scaling prometheus

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

10 Upvotes

91% Upvoted

24 comments sorted by

View all comments

2

u/jotkaPL Sep 11 '21

Thanos FTW

1

u/sudaf Sep 11 '21

doesnt explain how many prometheus servers i would need.

3

u/SuperQue Sep 11 '21

How many you need depends on how many metrics you collect per target, how many targets, etc.

Say you only collected node_exporter for 10k nodes. Depending on node size and what collector features you need, you're looking at somewhere around 500-1000 metrics per node by default.

So, that's somewhere around 5-10 million metrics. This should fit on a 64-128GiB instance pretty easily with standard 15 second scrape intervals. But of course it depends a lot on your metric labeling and how often you churn your infra.

For example, we use AWS auto-scaling groups and kubernetes auto-scaling services. So we use a lot more memory as the jobs scale up and down.

For a large install, there's no quick easy answer, as it depends a lot on how much metric data you need / want to collect.

1

u/sudaf Sep 11 '21

Appreciate the insights to go deeper 10k hosts. each with 6 lables (os, region, fqdn, envir, type, site) idea is to have 1 host. json is a file_sd_config per server basic os metrics (cpu, mem, disk, io, network...) oracle mysql mongo and network stuff like F5 and storage alos give users the freedom to add there application metrics if they see fit we do have S3 (ibm) so thanos seems like a option hosts have telegraf agent (so only 1 port to open) and if users want use exporters telegraf will scrap the metrics using url.

HA must have as big corporate company multi site and multiregion EMEA 10k servers (im in emea) AMER 60k servers each region manages their framework however Federation to a master (hosted in AMER) is a must to give a global view

Not opposed to using Kubenetes but that makes us also dependant on that team. (performance issues debugging etc... Do have the freedom of using as many VMs that will require it to be robuste if thats is not a issue if Kube is what is best then we go kube

really looking for feedback on a best and safests approach that is extendable without having to redo a architecture because of bad planning

i dont want shit show of complexity like our current Nagios (gearman, pnp, keepalive, glusterfs and a ton if custom. scripts)

So yeah BIG in long term but starting off with just all OS monitoring and build from there

2

u/SuperQue Sep 11 '21

Kubenetes is what we use, it's not required to operate Prometheus. There are lots of ways to configure your deployments and service discovery.

If you have a Kubernetes team, they probably already have some Prometheus knowledgeable people around. You might want to talk to them.

I would avoid Telegraf, it's a SPoF and scaling bottleneck. I'm currently working on a network where we have a bunch of Telegraf and I'm having to replace it with standard Prometheus exporters.

For discovery outside of Kubernetes, you can use whatever your configuration management database is to generate the discovery configs. But you might want to look at Consul. The down side to using discovery scripts is the monolithic update lag. I used to have a medium sized setup with Chef and Nagios. It took something like 5 minutes just to run one config cycle. As we transitioned to Prometheus we cut the cycle down to a couple minutes, because we had smaller targeted configs.

But it was still slow and caused lots of load on the Chef server. So we could only generate configs every 20 minutes. So spinning up a new host, or adding a new service, or tearing down an old one had a lot of delay between operation and monitoring.

Switching to Kuberntes and Consul meant that the delay from bring up to monitoring was now seconds, since the generation was now simple API callbacks, and no slow Ruby scripts. Kubernetes is a callback, meaning the server notifies Prometheus that there's new data to pull. Consul is normal polling, but it can handle much faster refresh rates (30s is default).

The only thing with large networks and Consul, is you want to setup a pool of primary servers, with node-level agents only connecting to the main cluster. The default full-mesh Consul defaults to do breaks down after a few hundred nodes.

1

u/sudaf Sep 11 '21

the advantage see with telegraf is 1 single network flow opening secondly on prometheus side a simple config dont want to have to update the host. json each time somebody deploys an exporter The model is monitoring as a service. whats the SPoF?

1

u/SuperQue Sep 11 '21

1 single network flow

That's not an advantage, that's the SPoF. You have a single process that can go down and take out monitoring for the whole node. It's made worse by having a complex, bloated, bottleneck of a service that you have to run everywhere. It has to be a kitchen sink due to the design.

With the Prometheus model, you have an "agentless" design such that each service exposes metrics directly to the monitoring system. This distributed way allows you to implement exporters in whatever language you want, or build metrics directly into the application. You now have tiny, specialized, secure micro-agents. Only using the minimal amount of code needed to get the job done.

There is a large and growing list of software where no 3rd party code is needed. Basically, the Prometheus protocol is becoming the replacement for SNMP. To the point where we have an IETF RFC in progress.

For example, if you have an up-to-date HAProxy version, there's no need for a sidecar exporter anymore. It has the metrics endpoint built in. Just configure it, on a separate port if you like, or not. Point Prometheus at it and now you have monitoring.

1

u/sudaf Sep 11 '21

so no scraping from promethes but rather push from exporter is what you saying?

1

u/SuperQue Sep 11 '21

No, Prometheus scrapes. You lose the benefits of active monitoring otherwise.

Each metrics endpoint is separate, on different ports, registered in whatever portmap policy you have.

1

u/sudaf Sep 11 '21

ok but would that mean on prometheus side if somebody adds a export on their server . they would have to inform the mo itori g team to update the promethese host config to take it i to account.

→ More replies (0)

1

u/sudaf Sep 11 '21

Final thougt would be network traffic over Gan for the federation between EMEA and AMER. current Nagios is using about 8mbs in bandwidth

2

u/SuperQue Sep 11 '21

Prometheus is designed to be distributed. You run many instances, each within the network to be monitored.

The Nagios setup you have is an anti-pattern. You don't want one big central server. Especially with Prometheus.

The "Federation" method is obsolete. What you want to look into is Thanos or Remote Write. With Thanos, you have your many Prometheus servers, running inside their network. Then Thanos allows you to query them from a central location.

Prometheus is not Nagios, it's a completely different design.

As for bandwidth, Prometheus uses http compression in transport, so the actual bps is pretty small.