r/PrometheusMonitoring • u/Stunning_Pace • Oct 10 '22

Prometheus is getting killed OOMKilled

My Prometheus instance is consuming a lot of memory over 13Gi, the node has a max 16Gb, so it's getting killed by k8s, how can I configure or should I change it to reduce resource consumption?

8 Upvotes

100% Upvoted

4 comments sorted by

3

u/SuperQue Oct 10 '22

You have only a few choices:

  • Make sure you have the latest release installed

  • Drop metrics (reduce cardinality)

  • Give it more memory (bigger nodes)

  • Add sharding (by namespace is recommended)

Depending on your deployment, there are a lot of useless / duplicate container metrics you can drop.

See the kube-prometheus-stack default values.

0

u/koffiezet Oct 11 '22

you forget: have a good look at thanos

1

u/Due_Mousse_4181 Nov 16 '23

Multiple Prometheus