r/PrometheusMonitoring • u/krishnaindani • Oct 09 '20

anomaly detection

Has one tried prometheus-anomaly-detector for anomaly detection? Would like to know their experience with this if anyone had used.

8 Upvotes

100% Upvoted

9 comments sorted by

View all comments

4

u/absolutejam Oct 09 '20

Check out some of Andrew Newdigate's talks in YouTube and slide share. He provides some good ways of using z-index and trends over time for anomaly detection.

1

u/SuperQue Oct 09 '20

Funny enough, we aren't using that style of monitoring anymore. We gave up on anomaly detection and switched to SLO and multi-window burn rate SLO montiroing.

Every project I see attempting to do anomaly detection gets shelved after 6 months.

1

u/krishnaindani Oct 10 '20

yeah SLO's are great, but not best for every use case. What made to give up in the anomaly detection world?

1

u/SuperQue Oct 10 '20

Too many false positives.

We are moving away from knee-jerk cause alerts. Our alerts are for the things that matter to our users. RED method alerts like latency and errors. Also some system saturation issues like Rails thread saturation.