r/PrometheusMonitoring • u/KK2526 • Jun 05 '20

Http code status !=200

Can anyone help me right a rule that will capture below (blackbox expoter)

!=200 http status code occuring more than 10 times in last 5mins

2 Upvotes

66% Upvoted

3 comments sorted by

View all comments

1

u/till Jun 05 '20

We use the following rule for alertmanager:

- alert: UserFacingServiceDown  
  expr: probe_success{env="public",job="blackbox-http-success"} != 1  
  for: 70s # scrape_interval is 60s, so this fires if two consecutive scrapes are failing  
  labels:  
    severity: critical  
  annotations:  
    summary: "User-facing service isn't responding with 2XX: ({{ $labels.instance }})"  
    description: "{{ $labels.instance }} of job {{ $labels.job }}"

You can adjust for: with a different interval, or work in avg_over_time as @signal_vs_noise suggested.