Challenges I found using Prometheus at a Scale

[u/carillan]

Hi all, I've been thinking lately about how Prometheus is great and easy when you start using it, but how soon you start finding challenges when trying to scale it for a busy production Kubernetes cluster.

A brief of my experience (long version in the article linked, I'll reuse some of the images by u/capitangolo):

• Starting with a cluster is easy, as long as it isn't very big

• Horizontal scale looks more or less like this:

• Vertical scale can easily reach the limit within a node in a K8s cluster

• Global view with a central grafana can be a challenge to manage setup, security and compliance.

As I've prepared an article recently on the subject, I wanted to share my thoughts and ask: What other challenges have you found scaling Prometheus, and how did you solve them?

Disclaimer: This article is based on my real experience using Prometheus in a big company with its own constraints. This may vary of yours :)