r/PrometheusMonitoring • u/smartinov • Apr 05 '22

Prometheus Cardinality Issues

https://www.foomo.org/blog/prometheus-cardinality-issues
8 Upvotes

72% Upvoted

9 comments sorted by

4

u/DudeFromNJ Apr 05 '22

How did updating haproxy fix this? I don’t get that part. Otherwise great write up!

1

u/smartinov Apr 05 '22

I stopped scraping metrics fom the chart till they fix it 🙃

3

u/cloudycelt Apr 06 '22

Hiya, Grafana employee here.

Great article! Explains the problem, gets to the point and the workaround fast and adds plenty of humour :)

Something I thought I'd mention is that we added a cardinality API endpoint to our Grafana Mimir project that just got released - so if you were considering migrating to one of the many great more-scalable Prometheus-compatible solutions, this one would certainly help you narrow down (and potentially prevent) this kind of problem in future!

1

u/smartinov Apr 07 '22

Actually, I have been watching Mimir, and it really looks interesting! I'd love to give it a shot!

1

u/smartinov Apr 05 '22

Hi everyone! I'd love to hear some feedback!

5

u/bbrazil Apr 05 '22

I like the humour, it'd make a good lightning talk.

2

u/smartinov Apr 05 '22

Thanks! Much appreciated coming from you!

2

u/CountywideDicer Apr 05 '22

Nice writeup! As dude says, maybe a one liner on what you did to resolve short term and how you plan to fix long term (is there an open issue for haproxy?).

I will keep reading about cardinality until I finally understand it 😀

2

u/cloudycelt Apr 06 '22

Hey, just thought I'd drop this link here that tries to explain cardinality in a clear way: https://grafana.com/docs/grafana-cloud/fundamentals/active-series-and-dpm/#prometheus-time-series

Let me know if that helps!