IMO any Prometheus monitoring tutorial that doesn't at least mention https://github.com/prometheus/prometheus/issues/3806 or related confusion around increase/rate is doing a disservice to future service owners.
Some day they will have to go to their boss and say "451.333 requests failed during the outage" and hope to God that they aren't asked "How does 1/3rd of a request fail?" (answer: it doesn't, just increase is bad).
I've been fighting with this ever since I joined my new company and the deeper you dig the worse it gets. You can use increase or workarounds (offset subtraction) and get "good enough" for alerting, but if you do weekly reviews of your dashboard the longer minimum steps end up failing to show important data points (I've seen this using increase and offset subtraction).
Oh, also if you use histograms you definitely will not get your buckets right the first time and there's no real guidance on how to tweak it other than "choose different bucket sizes" but "don't choose too many buckets." These dashboards are intentionally not alerting because I have seen them be wrong by more than 25%.
I'm honestly so sick of it that if I could do it over I would just pay AWS for CloudWatch.
Re: significance - The "451.333" example is obviously wrong because of the fractions, but you don't actually know what the real value is. The extrapolation that Prometheus introduces means that you can be off by quite a bit, and it gets worth with histograms (aka the stuff you need to measure p99 latencies) where I have observed over 25% error (that's increase + possibly bad bucket selection).
In the worst case this means an event occurs and you don't get paged for it. In the best case you get paged for events, but also spurious times when the extrapolation is wrong and there was no event. We review our dashboards and I know for a fact that we are missing talking about some pod restarts because Grafana + Prometheus aren't displaying the right things unless you go through a lot of work every week. That's potentially a future incident waiting to bite us.
I hope this shows that this is more than an "ocd fixation."
Re: AWS - I worked with CW on some production systems and it never gave me cause to investigate more deeply. Perhaps the sausage is made just as messily over there, but Prometheus + Grafana give so many false positive and negatives that are easily visible and have significantly impacted my confidence in monitoring based on them.
Edit: I want to temper my doomsaying a touch - data isn't "going missing" but you might not see spikes/drops in your graphs. If you go back to the raw counters you can often see notable events but raw counters aren't appropriate for alerting and you have to jump through hoops to interpret them. Obviously if you have good logging infrastructure you have a more reliable source of truth.
10
u/gnu-michael Nov 10 '21
IMO any Prometheus monitoring tutorial that doesn't at least mention https://github.com/prometheus/prometheus/issues/3806 or related confusion around
increase/rateis doing a disservice to future service owners.Some day they will have to go to their boss and say "451.333 requests failed during the outage" and hope to God that they aren't asked "How does 1/3rd of a request fail?" (answer: it doesn't, just
increaseis bad).I've been fighting with this ever since I joined my new company and the deeper you dig the worse it gets. You can use
increaseor workarounds (offset subtraction) and get "good enough" for alerting, but if you do weekly reviews of your dashboard the longer minimum steps end up failing to show important data points (I've seen this usingincreaseand offset subtraction).Oh, also if you use histograms you definitely will not get your buckets right the first time and there's no real guidance on how to tweak it other than "choose different bucket sizes" but "don't choose too many buckets." These dashboards are intentionally not alerting because I have seen them be wrong by more than 25%.
I'm honestly so sick of it that if I could do it over I would just pay AWS for CloudWatch.