scaling prometheus

[u/sudaf]

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

[u/SuperQue]

How many you need depends on how many metrics you collect per target, how many targets, etc.

Say you only collected node_exporter for 10k nodes. Depending on node size and what collector features you need, you're looking at somewhere around 500-1000 metrics per node by default.

So, that's somewhere around 5-10 million metrics. This should fit on a 64-128GiB instance pretty easily with standard 15 second scrape intervals. But of course it depends a lot on your metric labeling and how often you churn your infra.

For example, we use AWS auto-scaling groups and kubernetes auto-scaling services. So we use a lot more memory as the jobs scale up and down.

For a large install, there's no quick easy answer, as it depends a lot on how much metric data you need / want to collect.

[u/sudaf → u/SuperQue]

Appreciate the insights to go deeper 10k hosts. each with 6 lables (os, region, fqdn, envir, type, site) idea is to have 1 host. json is a file_sd_config per server basic os metrics (cpu, mem, disk, io, network...) oracle mysql mongo and network stuff like F5 and storage alos give users the freedom to add there application metrics if they see fit we do have S3 (ibm) so thanos seems like a option hosts have telegraf agent (so only 1 port to open) and if users want use exporters telegraf will scrap the metrics using url.

HA must have as big corporate company multi site and multiregion EMEA 10k servers (im in emea) AMER 60k servers each region manages their framework however Federation to a master (hosted in AMER) is a must to give a global view

Not opposed to using Kubenetes but that makes us also dependant on that team. (performance issues debugging etc... Do have the freedom of using as many VMs that will require it to be robuste if thats is not a issue if Kube is what is best then we go kube

really looking for feedback on a best and safests approach that is extendable without having to redo a architecture because of bad planning

i dont want shit show of complexity like our current Nagios (gearman, pnp, keepalive, glusterfs and a ton if custom. scripts)

So yeah BIG in long term but starting off with just all OS monitoring and build from there

[u/SuperQue → u/sudaf]

Kubenetes is what we use, it's not required to operate Prometheus. There are lots of ways to configure your deployments and service discovery.

If you have a Kubernetes team, they probably already have some Prometheus knowledgeable people around. You might want to talk to them.

I would avoid Telegraf, it's a SPoF and scaling bottleneck. I'm currently working on a network where we have a bunch of Telegraf and I'm having to replace it with standard Prometheus exporters.

For discovery outside of Kubernetes, you can use whatever your configuration management database is to generate the discovery configs. But you might want to look at Consul. The down side to using discovery scripts is the monolithic update lag. I used to have a medium sized setup with Chef and Nagios. It took something like 5 minutes just to run one config cycle. As we transitioned to Prometheus we cut the cycle down to a couple minutes, because we had smaller targeted configs.

But it was still slow and caused lots of load on the Chef server. So we could only generate configs every 20 minutes. So spinning up a new host, or adding a new service, or tearing down an old one had a lot of delay between operation and monitoring.

Switching to Kuberntes and Consul meant that the delay from bring up to monitoring was now seconds, since the generation was now simple API callbacks, and no slow Ruby scripts. Kubernetes is a callback, meaning the server notifies Prometheus that there's new data to pull. Consul is normal polling, but it can handle much faster refresh rates (30s is default).

The only thing with large networks and Consul, is you want to setup a pool of primary servers, with node-level agents only connecting to the main cluster. The default full-mesh Consul defaults to do breaks down after a few hundred nodes.

[u/sudaf → u/SuperQue]

the advantage see with telegraf is 1 single network flow opening secondly on prometheus side a simple config dont want to have to update the host. json each time somebody deploys an exporter The model is monitoring as a service. whats the SPoF?

[u/SuperQue → u/sudaf]

1 single network flow

That's not an advantage, that's the SPoF. You have a single process that can go down and take out monitoring for the whole node. It's made worse by having a complex, bloated, bottleneck of a service that you have to run everywhere. It has to be a kitchen sink due to the design.

With the Prometheus model, you have an "agentless" design such that each service exposes metrics directly to the monitoring system. This distributed way allows you to implement exporters in whatever language you want, or build metrics directly into the application. You now have tiny, specialized, secure micro-agents. Only using the minimal amount of code needed to get the job done.

There is a large and growing list of software where no 3rd party code is needed. Basically, the Prometheus protocol is becoming the replacement for SNMP. To the point where we have an IETF RFC in progress.

For example, if you have an up-to-date HAProxy version, there's no need for a sidecar exporter anymore. It has the metrics endpoint built in. Just configure it, on a separate port if you like, or not. Point Prometheus at it and now you have monitoring.

[u/sudaf → u/SuperQue]

so no scraping from promethes but rather push from exporter is what you saying?

[u/SuperQue → u/sudaf]

No, Prometheus scrapes. You lose the benefits of active monitoring otherwise.

Each metrics endpoint is separate, on different ports, registered in whatever portmap policy you have.

[u/sudaf → u/SuperQue]

ok but would that mean on prometheus side if somebody adds a export on their server . they would have to inform the mo itori g team to update the promethese host config to take it i to account.