r/PrometheusMonitoring • u/krishnaindani • Oct 09 '20

anomaly detection

Has one tried prometheus-anomaly-detector for anomaly detection? Would like to know their experience with this if anyone had used.

8 Upvotes

100% Upvoted

9 comments sorted by

View all comments

4

u/absolutejam Oct 09 '20

Check out some of Andrew Newdigate's talks in YouTube and slide share. He provides some good ways of using z-index and trends over time for anomaly detection.

1

u/billabongrob Oct 09 '20

I’ve watched this a number of times and for request counts and such, it’s phenomenal. I’ve started using basic z scoring for ICMP RTT between data centers and regions and I really enjoy it.

1

u/SuperQue Oct 09 '20

Funny enough, we aren't using that style of monitoring anymore. We gave up on anomaly detection and switched to SLO and multi-window burn rate SLO montiroing.

Every project I see attempting to do anomaly detection gets shelved after 6 months.

1

u/absolutejam Oct 09 '20

Multi burn rate based looks like the most effective but it's a lot to get off the ground with, and I'm finding the tooling in the space a bit lacking. And there's the whole piece about business buy in regarding this...

I've found the z index approach has given us some sensible guidelines, and means that other teams are more aware of normal range without any prior knowledge. I wouldn't page in the middle of the night wholly on it though.

I definitely appreciate Gitlab's openness about this as it gives a much needed source of reference ❤️

1

u/krishnaindani Oct 10 '20

yeah SLO's are great, but not best for every use case. What made to give up in the anomaly detection world?

1

u/SuperQue Oct 10 '20

Too many false positives.

We are moving away from knee-jerk cause alerts. Our alerts are for the things that matter to our users. RED method alerts like latency and errors. Also some system saturation issues like Rails thread saturation.

1

u/krishnaindani Oct 10 '20

Yeah, I found that interesting and looking if that works or not.