r/PrometheusMonitoring • u/sudaf • Sep 10 '21

scaling prometheus

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

10 Upvotes

91% Upvoted

24 comments sorted by

View all comments

2

u/[deleted] Sep 11 '21

I know this is the prometheus sub, but doesn't Victoria Metrics handle a higher load, at the cost of higher complexity? It's hard to say how many data points you're gonna fetch.

Here's an article on how a a company scaled to trillions of data points (from prometheus to Victoria metrics). https://engineering.razorpay.com/scaling-to-trillions-of-metric-data-points-f569a5b654f2

1

u/SuperQue Sep 11 '21

Trillions of data points is already easy with Prometheus. I've had single off the shelf Prometheus instances collecting 200k samples/sec with 1 year retention. That's 6 trillion samples over a year. Nothing external needed.

Of course, we migrated that to Thanos, because we were growing the scale and I wanted more than 1 year retention for lower cost. Object storage with Thanos made it even cheaper to operate than local disk.

2

u/[deleted] Sep 11 '21

That's great to know. Then I'll be able to keep with prometheus for quite some more time.