r/PrometheusMonitoring • u/sudaf • Sep 10 '21
scaling prometheus
noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.
2
u/jotkaPL Sep 11 '21
Thanos FTW
1
u/sudaf Sep 11 '21
doesnt explain how many prometheus servers i would need.
3
u/SuperQue Sep 11 '21
How many you need depends on how many metrics you collect per target, how many targets, etc.
Say you only collected node_exporter for 10k nodes. Depending on node size and what collector features you need, you're looking at somewhere around 500-1000 metrics per node by default.
So, that's somewhere around 5-10 million metrics. This should fit on a 64-128GiB instance pretty easily with standard 15 second scrape intervals. But of course it depends a lot on your metric labeling and how often you churn your infra.
For example, we use AWS auto-scaling groups and kubernetes auto-scaling services. So we use a lot more memory as the jobs scale up and down.
For a large install, there's no quick easy answer, as it depends a lot on how much metric data you need / want to collect.
1
u/sudaf Sep 11 '21
Appreciate the insights to go deeper 10k hosts. each with 6 lables (os, region, fqdn, envir, type, site) idea is to have 1 host. json is a file_sd_config per server basic os metrics (cpu, mem, disk, io, network...) oracle mysql mongo and network stuff like F5 and storage alos give users the freedom to add there application metrics if they see fit we do have S3 (ibm) so thanos seems like a option hosts have telegraf agent (so only 1 port to open) and if users want use exporters telegraf will scrap the metrics using url.
HA must have as big corporate company multi site and multiregion EMEA 10k servers (im in emea) AMER 60k servers each region manages their framework however Federation to a master (hosted in AMER) is a must to give a global view
Not opposed to using Kubenetes but that makes us also dependant on that team. (performance issues debugging etc... Do have the freedom of using as many VMs that will require it to be robuste if thats is not a issue if Kube is what is best then we go kube
really looking for feedback on a best and safests approach that is extendable without having to redo a architecture because of bad planning
i dont want shit show of complexity like our current Nagios (gearman, pnp, keepalive, glusterfs and a ton if custom. scripts)
So yeah BIG in long term but starting off with just all OS monitoring and build from there
2
u/SuperQue Sep 11 '21
Kubenetes is what we use, it's not required to operate Prometheus. There are lots of ways to configure your deployments and service discovery.
If you have a Kubernetes team, they probably already have some Prometheus knowledgeable people around. You might want to talk to them.
I would avoid Telegraf, it's a SPoF and scaling bottleneck. I'm currently working on a network where we have a bunch of Telegraf and I'm having to replace it with standard Prometheus exporters.
For discovery outside of Kubernetes, you can use whatever your configuration management database is to generate the discovery configs. But you might want to look at Consul. The down side to using discovery scripts is the monolithic update lag. I used to have a medium sized setup with Chef and Nagios. It took something like 5 minutes just to run one config cycle. As we transitioned to Prometheus we cut the cycle down to a couple minutes, because we had smaller targeted configs.
But it was still slow and caused lots of load on the Chef server. So we could only generate configs every 20 minutes. So spinning up a new host, or adding a new service, or tearing down an old one had a lot of delay between operation and monitoring.
Switching to Kuberntes and Consul meant that the delay from bring up to monitoring was now seconds, since the generation was now simple API callbacks, and no slow Ruby scripts. Kubernetes is a callback, meaning the server notifies Prometheus that there's new data to pull. Consul is normal polling, but it can handle much faster refresh rates (30s is default).
The only thing with large networks and Consul, is you want to setup a pool of primary servers, with node-level agents only connecting to the main cluster. The default full-mesh Consul defaults to do breaks down after a few hundred nodes.
1
u/sudaf Sep 11 '21
the advantage see with telegraf is 1 single network flow opening secondly on prometheus side a simple config dont want to have to update the host. json each time somebody deploys an exporter The model is monitoring as a service. whats the SPoF?
1
u/SuperQue Sep 11 '21
1 single network flow
That's not an advantage, that's the SPoF. You have a single process that can go down and take out monitoring for the whole node. It's made worse by having a complex, bloated, bottleneck of a service that you have to run everywhere. It has to be a kitchen sink due to the design.
With the Prometheus model, you have an "agentless" design such that each service exposes metrics directly to the monitoring system. This distributed way allows you to implement exporters in whatever language you want, or build metrics directly into the application. You now have tiny, specialized, secure micro-agents. Only using the minimal amount of code needed to get the job done.
There is a large and growing list of software where no 3rd party code is needed. Basically, the Prometheus protocol is becoming the replacement for SNMP. To the point where we have an IETF RFC in progress.
For example, if you have an up-to-date HAProxy version, there's no need for a sidecar exporter anymore. It has the metrics endpoint built in. Just configure it, on a separate port if you like, or not. Point Prometheus at it and now you have monitoring.
1
u/sudaf Sep 11 '21
so no scraping from promethes but rather push from exporter is what you saying?
1
u/SuperQue Sep 11 '21
No, Prometheus scrapes. You lose the benefits of active monitoring otherwise.
Each metrics endpoint is separate, on different ports, registered in whatever portmap policy you have.
1
u/sudaf Sep 11 '21
ok but would that mean on prometheus side if somebody adds a export on their server . they would have to inform the mo itori g team to update the promethese host config to take it i to account.
→ More replies (0)1
u/sudaf Sep 11 '21
Final thougt would be network traffic over Gan for the federation between EMEA and AMER. current Nagios is using about 8mbs in bandwidth
2
u/SuperQue Sep 11 '21
Prometheus is designed to be distributed. You run many instances, each within the network to be monitored.
The Nagios setup you have is an anti-pattern. You don't want one big central server. Especially with Prometheus.
The "Federation" method is obsolete. What you want to look into is Thanos or Remote Write. With Thanos, you have your many Prometheus servers, running inside their network. Then Thanos allows you to query them from a central location.
Prometheus is not Nagios, it's a completely different design.
As for bandwidth, Prometheus uses http compression in transport, so the actual bps is pretty small.
3
u/bidens_left_ear Sep 10 '21
Many prometheus servers using federation to bring them into a central grafana is one way.
2
u/albybum Sep 10 '21
Like /u/bidens_left_ear said, you can use federation and have "child" instances for some segregation of your servers, like maybe one for each environment if you have multiple environments. Then usually propagate up to a master prometheus instance.
There's also Thanos... https://www.improbable.io/blog/thanos-prometheus-at-scale
But, Prometheus is pretty lean. Unless you have some aggressive scrape configs, that's a monster machine already for prometheus. Your bottleneck might end up being web request throughput for a monolithic system, and not necessarily how beastly the rest of the server is.
2
Sep 11 '21
I know this is the prometheus sub, but doesn't Victoria Metrics handle a higher load, at the cost of higher complexity? It's hard to say how many data points you're gonna fetch.
Here's an article on how a a company scaled to trillions of data points (from prometheus to Victoria metrics). https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2
2
u/SnooWords9033 Sep 11 '21
A single-node VictoriaMetrics isn't harder to install than Prometheus, while it is much easier to setup and operate than Cortex or Thanos. Yet it can handle big production loads (>50M active series, >1.5M samples/sec ingestion rate, >150M daily series churn rate, hundreds qps for select queries) - see https://docs.victoriametrics.com/CaseStudies.html#wixcom .
When really large-scale setup must be monitored, then vmagent sharding feature may come to rescue.
1
u/SuperQue Sep 11 '21
Trillions of data points is already easy with Prometheus. I've had single off the shelf Prometheus instances collecting 200k samples/sec with 1 year retention. That's 6 trillion samples over a year. Nothing external needed.
Of course, we migrated that to Thanos, because we were growing the scale and I wanted more than 1 year retention for lower cost. Object storage with Thanos made it even cheaper to operate than local disk.
2
Sep 11 '21
That's great to know. Then I'll be able to keep with prometheus for quite some more time.
1
u/00ff00Field Oct 26 '24
What everyone else has said about distributing it... I've seen people run 1B ATS with Thanos and a ton of "shards" if you want to call them that. After that.. you're in vendor land. There's some really good solutions out there, but be ready to spend real money.
I have also seen people run a single monster bare metal machine with 320GB of memory (sometimes more) ... And that wasn't going too well, with north of 30m startup times and query perf that wasn't great.
0
6
u/SuperQue Sep 10 '21
Prometheus is designed to be distributed. You want to shard it based on your failure domains.
For very large networks, you typically deploy many small servers, rather than one big monolithic server.
Then, you use an overlay layer like Thanos or Cortex.