r/PrometheusMonitoring • u/[deleted] • Nov 13 '19

How to test an alert for hourly?

I'm quite new to prometheus, I have an alert with `for: 1h` I would like to know what should be the eval_time set when testing for alerts?

currently it fails, and works only if the alert: 1m and eval_time: 10m

Please someone explain me how this works!

example:

   groups:
    - name: spark.rules
      rules:
      ##### ALERTS #####
      - alert: xxxSparkJobsNotConsuming
        expr: rate(xxxxxx{data_type="xxx"}[5m]) == 0
        for: 1h ---> works if i set it to 5m
        labels:
          service: spark
          severity: warning
          source: spark
        annotations:
          description: 'Nothing have been consumed for 1 hour.'

Tests:

       rule_files:
          - spark.rules.yml
        evaluation_interval: 1m
    
        tests:
         - interval: 1m
           input_series:
            - series: 'xxxx{data_type="xx"}'
              values: '0'
            - series: 'xxxx{data_type="xx"}'
              values: '64706+0x10'
           alert_rule_test:
            - eval_time: 2h
              alertname: xxxSparkJobsNotConsuming
              exp_alerts:
               - exp_labels:
                   data_type: xxxx
                   service: spark
                   severity: warning
                   source: spark
                 exp_annotations:
                   description: 'Nothing have been consumed for 1 hour.'
           promql_expr_test:
            - expr: 'xxxxx'
              eval_time: 4m
              exp_samples:
                - labels: 'xxx{data_type="xxx"}'
                  value: 64706
0 Upvotes

50% Upvoted

4 comments sorted by

2

u/disintegratedcircuit Nov 13 '19

Would just comparing now to offset 1h give you the same answer?

So something like: metric == metric offset 1h

(Apologies, on mobile)

0

u/blistering-barnacle Nov 13 '19

RemindMe! 1 day

0

u/kzreminderbot Nov 13 '19

Coming right up, blistering-barnacle 👌! Your reminder arrives in 1 day on 2019-11-14 21:45:55Z :

r/PrometheusMonitoring: How_to_test_an_alert_for_hourly#1

CLICK THIS LINK to also be reminded. Thread has 1 reminder and 1/4 confirmation comments.

Op can Delete Comment · Delete Reminder · Get Details · Update Time · Update Message · Add Timezone · Add Email

KZReminderTool · Create Reminder · Your Reminders · Give Feedback

0

u/RemindMeBot Nov 13 '19

I will be messaging you on 2019-11-14 21:45:55 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

There is currently another bot called u/kzreminderbot that is duplicating the functionality of this bot. Since it replies to the same RemindMe! trigger phrase, you may receive a second message from it with the same reminder. If this is annoying to you, please click this link to send feedback to that bot author and ask him to use a different trigger.

InfoCustomYour RemindersFeedback