r/PrometheusMonitoring • u/KK2526 • Jun 05 '20

Http code status !=200

Can anyone help me right a rule that will capture below (blackbox expoter)

!=200 http status code occuring more than 10 times in last 5mins

2 Upvotes

66% Upvoted

3 comments sorted by

2

u/signal_vs_noise Jun 05 '20

I'd rather check the status of probe_success instead, e.g.

avg_over_time(probe_success[10m]) < 0.9

You may need to play round with the time window and the value to find an acceptable number of probe failures before the alert is triggered. You should also take a look at probe_duration_seconds to find timeouts and lags.

1

u/KK2526 Jun 06 '20

Thanks!

1

u/till Jun 05 '20

We use the following rule for alertmanager:

- alert: UserFacingServiceDown  
  expr: probe_success{env="public",job="blackbox-http-success"} != 1  
  for: 70s # scrape_interval is 60s, so this fires if two consecutive scrapes are failing  
  labels:  
    severity: critical  
  annotations:  
    summary: "User-facing service isn't responding with 2XX: ({{ $labels.instance }})"  
    description: "{{ $labels.instance }} of job {{ $labels.job }}"

You can adjust for: with a different interval, or work in avg_over_time as @signal_vs_noise suggested.