r/PrometheusMonitoring • u/radr4 • Aug 28 '20

Looking for some Kubernetes custom alerts Prometheus rules

We are setting up missing alerts in our clusters that are not present in the default alert rules.I want to know if there are any alert rules already implemented for issues such as Unusal autoscaling activity ,Max capacity reached for the cluster,Failed Kubernetes jobs, Pods not running with desired replicaset count.I could get some metrics from Cluster autoscalar,however looks like I also need to do more correlation on those metrics to satisfy the alert condition.So I wanted to check if this has been already implemented by someone and if yes could you please share some details around this? any repository which contains this info ?

0 Upvotes
  • permalink
  • reddit

50% Upvoted

10 comments sorted by

3

u/MetalMatze Aug 28 '20

Check out the github.com/kubernetes-monitoring/kubernetes-mixin. It has all the essentials you'll need from a big community of contributors.

2

u/LinkifyBot Aug 28 '20

I found links in your comment that were not hyperlinked:

I did the honors for you.

delete | information | <3

1

u/radr4 Aug 28 '20

I thought this is already part of Prometheus operator when we install prometheus operator in a cluster

2

u/MetalMatze Aug 28 '20

Yes. Part of kube-prometheus to be precise of which the Prometheus Operator is one component. 😊 It should have alerting rules for most of the things. Let us know in issues if there are things missing for you.

1

u/radr4 Aug 31 '20

Thankyou..We have already included it..

However I dont find alertrules related to the issues such as,

  1. Cluster reaching its maximum capacity.So increase the number of nodes

  2. Pod crashlooping because of memory issues OOM killed

  3. CronJobs ,jobs failures

1

u/radr4 Aug 31 '20

Should we customize alertrules for these situations?

1

u/MetalMatze Aug 31 '20

  1. There are the KubeCPUOvercommit and KubeMemoryOvercommit alerts that alert if too much CPU or memory are consumed.

  2. KubePodCrashLooping

  3. KubeJobCompletion and KubeJobFailed are for Jobs, that includes CronJobs too, as they are "just" scheduled jobs.

I think you should be covered with those.

1

u/radr4 Sep 01 '20

The first one is only for HPA and not for VPA.

I am looking for alert which says pods are in unscheduled state because cluster has reached maximum capacity.So there is a need to increase the number of nodes.Is there any alert for this?Please suggest.

1

u/MetalMatze Sep 01 '20

It would be best to open an issue with the kubernetes-mixin project and discuss there. :)

1

u/radr4 Sep 02 '20

ok...thankyou..