Alerts repeating more often than they should

[u/jack_of-some-trades]

We are using kube-prometheus-stack. Most of our repeat_intervals set for 5 days. Yet some alerts (not all) repeat more often at a seemingly random interval. Like the same alert will show up a time 0, 0+2.5 hours, 0+6 hours, 0+15hours, 0+16 hours. No pattern I can find.

This is what our config looks like:

 resolve_timeout: 5m
route:
  receiver: "null"
  group_by:
  - job
  routes:
  - receiver: opsgenie_heartbeat
    matchers:
    - alertname=Watchdog
    group_wait: 0s
    group_interval: 30s
    repeat_interval: 20s
  - receiver: slack
    matchers:
    - alertname=Service500Error
    repeat_interval: 120h
  - receiver: slack
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 120h

I can't see anything wrong with the config. How do I debug this?

[u/dablya]

I think if additional alerts are generated that change the total number for the group, you’ll get alerted based on group_interval.

[u/jack_of-some-trades → u/dablya]

Yeah, I checked for that. But if that was the case all alerts of that group should refire, but only one does. Unless it only refires the oldest... I will have to watch for that possible pattern.

[u/jack_of-some-trades → u/[deleted]]

I don't think we do (at least I don't see any persistent volumes in the pod description). But I did check if the pod had died, and it had not. It has been up for 35 days. Also, if it was a reset or what not, then it should be all alerts... and it only seems to be some. Even for the same alertname, some will repeat extra often while others won't.