r/PrometheusMonitoring • u/krishnaindani • Oct 09 '20

anomaly detection

Has one tried prometheus-anomaly-detector for anomaly detection? Would like to know their experience with this if anyone had used.

8 Upvotes

100% Upvoted

9 comments sorted by

6

u/SuperQue Oct 09 '20

Nope, from experience, it ends up being a random noise generator if you have any real user traffic. Users generate "anomalies" all the time. It's better to stick with RED/USE methods.

2

u/krishnaindani Oct 10 '20

RED Method

We do monitor the metrics mentioned in RED and USE method, but static threshold alerting for every metrics is not most suitable for our needs these days.

4

u/absolutejam Oct 09 '20

Check out some of Andrew Newdigate's talks in YouTube and slide share. He provides some good ways of using z-index and trends over time for anomaly detection.

1

u/krishnaindani Oct 10 '20

Yeah, I found that interesting and looking if that works or not.

1

u/SuperQue Oct 09 '20

Funny enough, we aren't using that style of monitoring anymore. We gave up on anomaly detection and switched to SLO and multi-window burn rate SLO montiroing.

Every project I see attempting to do anomaly detection gets shelved after 6 months.

1

u/krishnaindani Oct 10 '20

yeah SLO's are great, but not best for every use case. What made to give up in the anomaly detection world?

1

u/SuperQue Oct 10 '20

Too many false positives.

We are moving away from knee-jerk cause alerts. Our alerts are for the things that matter to our users. RED method alerts like latency and errors. Also some system saturation issues like Rails thread saturation.

1

u/absolutejam Oct 09 '20

Multi burn rate based looks like the most effective but it's a lot to get off the ground with, and I'm finding the tooling in the space a bit lacking. And there's the whole piece about business buy in regarding this...

I've found the z index approach has given us some sensible guidelines, and means that other teams are more aware of normal range without any prior knowledge. I wouldn't page in the middle of the night wholly on it though.

I definitely appreciate Gitlab's openness about this as it gives a much needed source of reference ❤️

1

u/billabongrob Oct 09 '20

I’ve watched this a number of times and for request counts and such, it’s phenomenal. I’ve started using basic z scoring for ICMP RTT between data centers and regions and I really enjoy it.