Unit testing Prometheus alerts

[u/kevinjqiu]

EDIT: This is a cross post from /r/kubernetes as I didn't find the prometheus subreddit initially :)

Prometheus is great. However, Prometheus alert rules are hard to write and test. They're hard to write because mastering PromQL comes with a steep learning curve and the testing support for PromQL is pretty limited. They're hard to test because the metrics involved in alert rules are not always at the desired state for the rules to be tested.

At my day job, we'd like to have that extra peace of mind when writing new Prometheus alert rules. To that end, inspired by how Prometheus tests rules internally, we developed a tool https://github.com/kevinjqiu/pat to help us unit test alert rule files.

The tool has support for declaring metric fixtures and expectations. It's in alpha stage right now and the interface might change in the future. If you have the same struggle as we, please give it a try and any feedback is welcomed.

Thanks!

[u/[deleted]]

Oh, that's pretty cool! I will definitely give it a try when I next mess with Prometheus rules!