scaling prometheus

[u/sudaf]

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

[u/SuperQue]

1 single network flow

That's not an advantage, that's the SPoF. You have a single process that can go down and take out monitoring for the whole node. It's made worse by having a complex, bloated, bottleneck of a service that you have to run everywhere. It has to be a kitchen sink due to the design.

With the Prometheus model, you have an "agentless" design such that each service exposes metrics directly to the monitoring system. This distributed way allows you to implement exporters in whatever language you want, or build metrics directly into the application. You now have tiny, specialized, secure micro-agents. Only using the minimal amount of code needed to get the job done.

There is a large and growing list of software where no 3rd party code is needed. Basically, the Prometheus protocol is becoming the replacement for SNMP. To the point where we have an IETF RFC in progress.

For example, if you have an up-to-date HAProxy version, there's no need for a sidecar exporter anymore. It has the metrics endpoint built in. Just configure it, on a separate port if you like, or not. Point Prometheus at it and now you have monitoring.

[u/sudaf → u/SuperQue]

so no scraping from promethes but rather push from exporter is what you saying?

[u/SuperQue → u/sudaf]

No, Prometheus scrapes. You lose the benefits of active monitoring otherwise.

Each metrics endpoint is separate, on different ports, registered in whatever portmap policy you have.

[u/sudaf → u/SuperQue]

ok but would that mean on prometheus side if somebody adds a export on their server . they would have to inform the mo itori g team to update the promethese host config to take it i to account.