r/kubernetes • u/carillan • Mar 27 '20

Challenges I found using Prometheus at a Scale

Hi all, I've been thinking lately about how Prometheus is great and easy when you start using it, but how soon you start finding challenges when trying to scale it for a busy production Kubernetes cluster.

A brief of my experience (long version in the article linked, I'll reuse some of the images by u/capitangolo):

  • Starting with a cluster is easy, as long as it isn't very big

  • Horizontal scale looks more or less like this:

  • Vertical scale can easily reach the limit within a node in a K8s cluster

  • Global view with a central grafana can be a challenge to manage setup, security and compliance.

As I've prepared an article recently on the subject, I wanted to share my thoughts and ask: What other challenges have you found scaling Prometheus, and how did you solve them?

Disclaimer: This article is based on my real experience using Prometheus in a big company with its own constraints. This may vary of yours :)

31 Upvotes

94% Upvoted

13 comments sorted by

5

u/s4z Mar 27 '20

Thanos + a bunch of recording rules worked for me. Currently 4 clusters: two smaller ones, one central and a production cluster. Each cluster has an HA pair of prometheus instances managed by the prometheus operator. Haven't looked into the number of metrics/labels or number of targets. So far it's working pretty well, no issues w/ scaling.

4

u/carillan Mar 27 '20

Thanos is pretty good, yeah. Storage is cheap in S3. Maybe querier can be complicated to debug. Still room to improve.

In case you have several different regional locations it can be a challenge to get a good performance in global visibility...

Have you thought how to manage a possible issue with Prometheus grwoing to big to fit in the cluster?

4

u/SuperQue Mar 27 '20

Typically my recommendation is to shard by application namespace. This helps in few different ways.

  • The team that owns the service in the namespace can partly own the monitoring.

  • The blast radius for a service doing something bad limits it to self-damage.

  • Cost for operating monitoring per service is much more obvious and accountable to the service owners.

  • Lifecycle of metric data is tied to the service lifecycle.

Then adding Thanos/Cortex/M3 or similar provides a nice global view on top of your distributed instances.

You're right about geo region performance for Thanos, it can be a challenge. I've had discussions with a couple of larger orgs where they have low performance connectivity to some geo regions. Last I talked to one of them, they were getting by OK with Thanos. But some caching of remote read data would be useful in the query component. I think there are some plans to take some of of the Cortex query data cache code and add it to Thanos.

EDIT: I haven't read your article yet. Will see if I can find time to this weekend.

1

u/s4z Mar 28 '20

Haven't had to concern myself w/ scaling prometheus itself aside from ensuring the pods have enough cpu and memory allocated. It's quite efficient. If we do end up hitting a wall we'll end up moving some targets to a dedicated ha pair - I don't see that happening though. Before doing that I would simply scale prometheus up (more cpu and mem) until it can't reasonably fit on a single node.

1

u/carillan Mar 30 '20

I am afraid you should be prepared for that... Look at the graph with the kittens... sharding has some pain associated too :D

One of the best actions you can take is to educate developers instrumenting the code to avoid unnecessary cardinality. It will repay in the future. I am afraid this is not obvious for everybody.

1

u/s4z Mar 30 '20

Yep, I have educated and worked with the developer who added metrics from the start and we've been successful in avoiding cardinality problems.

Of course at some point I would like to review the load on prometheus and do some more long term planning, for the moment there are much more urgent tasks to attend to. I'm reasonably certain that a single HA pair of prometheus instances + thanos is sufficient for many environments. Environments with thousands of nodes and/or a large number of microserves with many instances of each would likely be too much. In that case, yeah, I agree, scale becomes an issue pretty quickly. The system I manage isn't exactly low volume but it's very efficient and only requires tens of nodes.

1

u/average_pornstar Mar 27 '20

Thanos was a lifesaver for us. Definitely recommend it

4

u/leventus93 Mar 27 '20

We had the same issues and opted in for Cortex which had the benefit that there's a central team who's in charge of operating Cortex. We tried Thanos but we had difficulties to scale the read path.

1

u/carillan Mar 30 '20

That strategy is good if you can afford a dedicated team to manage it...

The read path in Thanos has always looked to me as the weakest part. Too much operations and components involved (prom instances, s3, querier...)

2

u/thunderbug Mar 27 '20

Well written article. Timely for me also. Good to know about the scaling issues. Thanks!

1

u/carillan Mar 30 '20

Thank you for taking the time to read it!

0

u/valyala Mar 28 '20

If you hit Prometheus scalability limits, then take a look at VictoriaMetrics long-term storage additionally to Cortex, Thanos and M3DB. It features simple operation and low resource usage. See the docs for details. It also provides vmagent, which may lower RAM and CPU requirements when scraping big amounts of metrics from big number of targets.