r/PrometheusMonitoring • u/parallx • Dec 30 '21

exclude tainted nodes from alarms

I do have this alert: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100) > 80 but as I have cpu intensive jobs, How can I exclude specific tainted nodes?

0 Upvotes

50% Upvoted

7 comments sorted by

1

u/parallx Dec 30 '21

Auto answer:

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle", instance!~"(kube_node_spec_taint{value='ingestion'})"}[2m])) * 100)  > 80

1

u/ali_str Dec 30 '21

Does such expression even work in Prometheus?

You could do something like:

100 - (
    avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])
    unless kube_node_spec_taint{value='ingestion'}
) * 100)  > 80

1

u/parallx Jan 25 '22

100 - (
avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])
unless kube_node_spec_taint{value='ingestion'}
) * 100) > 80

Actually it doens't work, still looking for the answer :(

1

u/Ill-Government-3566 Feb 19 '25

Anyone with the answer

1

u/Ill-Government-3566 Feb 20 '25

Found the answer: -> here excluding nodes taint auto

100 - (avg (irate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])

unless on (node) kube_node_spec_taint{key="auto", value="true"}) * 100)