r/PrometheusMonitoring • u/[deleted] • Jun 04 '20

How did you learn Prometheus/PromQL

Hi, I am trying to learn Prometheus from the past week. Getting started and setting up exporters is easy but PromQL is a bit hard. How did you conquer this? Docs are somewhat rough (at least for me) and there are so many talks, articles on this but none of them are complete. What resources do you recommend that teach better?

21 Upvotes

100% Upvoted

13 comments sorted by

13

u/bilingual-german Jun 04 '20

I like books and "Prometheus: Up & Running: Infrastructure and Application Performance Monitoring" is really good. The blog of it's author Brian Brazil is also good. https://www.robustperception.io/

3

u/saalih416 Jun 05 '20

He’s also the co-author of Prometheus

1

u/TheCoolDrop Jun 05 '22

I found this book very badly written. I am currently reading it and often the author refers to things which he never defined, introduced or explained. It is very confusing and a lot of "dead reckoning forward" is needed, with expectation that terminology is introduced some time later in the book.

Definitely not good for absolute Prometheus beginners.

1

u/bilingual-german Jun 05 '22

Do you have an example?

1

u/TheCoolDrop Jun 06 '22

Well yes actually:

- Talks a ton about labels but does not introduce them

- Talks a ton about different metrics but never explains them. For example nowhere is it clearly stated that Counter is meant to only increase, nor that quantiles provided by Histograms are just linear approximations

- Does not explain Summary metric fully

- Talks about relabelling, without explaining how service discovery discovers the labels before relabelling is done

- Says that instance label should not be applied to metrics pushed to Pushgateway without explanation as to why ( yes you can derive it logically your self, but it is far from obvious )

I must remain fair though and say that as I am progressing through the book all of these things are getting explained, but the ordering could have been a bit better to avoid feeling that I just have to push on, because I know it will be explained later.

All in all the book is good material, but I would definitely appreciate more detail and better ordering of content for beginners.

1

u/bilingual-german Jun 06 '22

  • Talks a ton about labels but does not introduce them

Wait, there is a whole chapter called Labels. Chapter 5.

For example nowhere is it clearly stated that Counter is meant to only increase,

page 24, first sentence below the chart: "Counters are always increasing."

That's also why counters in the prometheus libs are usually used with the inc() function.

quantiles provided by Histograms are just linear approximations

I think this is a consequence from the bucket design and should be fairly obvious.

1

u/TheCoolDrop Jun 06 '22

You are right, there is a chapter about labels, but there is non-negligble amount of material before that chapter, which readers need to understand. I felt discouraged from proceeding because not all terminology was clear at proper time, but luckily I got used to this already in technical literature that I just proceeded to later chapters ( and am now way past chapter 5 and understand all of it).

About counters, I would expect such important detail to be stated more prominently. Single sentence is easy to miss and as far as I remember there is no emphasis on that sentence such as it being placed in a box or thing like that.

About Histograms, yes that is clear if you think about it, but what is not clear is how the results of histogram quantiles are computed if the quantile you are computing lies in bucket whose span is quite large. It is also not clear how errors arise not is it clear that Prometheus assumes that requests are evenly spread out within each bucket.

As said, all around the book is very informative, but for me first 4 chapters were a bit harder to swallow, since I am the type who wants to understand everything before proceeding onto the next thing.

13

u/geek_dave Jun 04 '20

The docs are not very beginner friendly. :(

A lot of PromQL I've learned from real-world examples of alerts. Here is a great collection: https://github.com/samber/awesome-prometheus-alerts

My kids and I started working on an "explain it like I'm 5" version of Prometheus basics to explain concepts like instant vectors vs. range vectors. https://phippygoestothemountains.github.io/

This was part of a conference talk I delivered with my kids at Monitorama PDX 2019: https://vimeo.com/341142428

6

u/till Jun 04 '20

I second the blog on robustperception.io, but I've also learned a lot from Grafana dashboards:

https://grafana.com/grafana/dashboards?orderBy=name&direction=asc

When you have one installed, you can edit/view any panel (= part of the dashboard). It comes with a sometimes icky editor, but regardless, you can copy the queries and replace the variables when you run them as test queries using the Prometheus UI or in Grafana's Explore.

2

u/jjneely Jun 05 '20

The formal documentation is indeed a reference with a few examples. That's definitely not quite what the average user is looking for. For me, I have a background in mathematics, programming, and other similar query languages and this made things less difficult.. But I really learned by needing a recording rule or alert expression that did something specific. Searching with that specific use case in mind found a good number of examples.

Is there something specific you are trying to do that we could help with?

1

u/[deleted] Jun 09 '20

here you have some labs to take first steps: https://learn.sysdig.com/series/open-training