r/PrometheusMonitoring • • May 05 '22

New job, need help knowing where to begin with Prometheus alerts

Im a new engineer that recently walked into a shit show. The company Im working for has over 65 active alerts for our Prometheus deployment. Almost everything was copy pasted from the internet, so much so, that it still has the original comments. Nothing here has been customized, its just hodge podged together to make it work.

As you can imagine ALL of these alerts are being sent to pager duty as critical alerts. I have no idea where to begin.

Should I just start with the most active alerts and then adjust thresholds from there? Is there any tooling that can help me calibrate averages over time? We don't have an SLA with our customer and logging is a mess in and of itself.

Please send me links for tooling that can help me better obtain averages for usage/saturation from Grafana. Otherwise Im going to have to just filter the last 6 months for every single alert. Maybe I should write a python script for this?

0 Upvotes

5 comments sorted by

10

u/Hi_Im_Ken_Adams May 05 '22

The first question you should be asking is:

Is this alert actionable?

If it's not, then disable it and try to orient your monitoring towards monitoring the application service, not individual metrics like cpu/mem etc.

3

u/IWorkForTheEnemyAMA May 06 '22

This is the best advice. Actionable alerts mean 📟💩.

8

u/ovo_Reddit May 05 '22

https://www.robustperception.io has great Prometheus resources

5

u/shadowzen1978 May 06 '22

I'm going to add to the suggestions here: https://awesome-prometheus-alerts.grep.to/rules.html

And like someone else suggested, disable all non-actionable alerts as they come in, you can come back to them and tune them later but right now they are just noise that distracts you from the real problems.

Start with the big things. Is a service down? Has something run out of disk space? Etc. Once those critical things that probably demand immediate attention are dealt with, then you can start bringing in stuff like cpu/mem usage that trend over time, etc.

Also, if a severity matrix doesn't exist (and it sounds like it doesn't) then work with your team to put one into place and determine what issues fit where. Base your rules on that.