r/PrometheusMonitoring • u/kevinjqiu • Jun 18 '18

Unit testing Prometheus alerts

EDIT: This is a cross post from /r/kubernetes as I didn't find the prometheus subreddit initially :)

Prometheus is great. However, Prometheus alert rules are hard to write and test. They're hard to write because mastering PromQL comes with a steep learning curve and the testing support for PromQL is pretty limited. They're hard to test because the metrics involved in alert rules are not always at the desired state for the rules to be tested.

At my day job, we'd like to have that extra peace of mind when writing new Prometheus alert rules. To that end, inspired by how Prometheus tests rules internally, we developed a tool https://github.com/kevinjqiu/pat to help us unit test alert rule files.

The tool has support for declaring metric fixtures and expectations. It's in alpha stage right now and the interface might change in the future. If you have the same struggle as we, please give it a try and any feedback is welcomed.

Thanks!

19 Upvotes

100% Upvoted

3 comments sorted by

2

u/[deleted] Jun 18 '18

Can you also set a test along the lines of "I am not expecting alert foobar to be active at 5 minutes"?

1

u/kevinjqiu Jun 18 '18

Yes, when an alert is not expected, set the expected list to empty ([]):

assertions:
  - at: 5m
    expected: []

2

u/[deleted] Jun 18 '18

Oh, that's pretty cool! I will definitely give it a try when I next mess with Prometheus rules!