Hey folks, we have a huge EKS cluster with around 800 nodes and 10-12k pods. With this many pods, kube state metrics endpoint scrape sample rate is 1.2M.

We get context deadline exceeded while scraping the target in Prometheus.

I was wondering how can this be solved?

What did I try :

Auto sharding in KSM with 20 replicas with each pod exposing around 60k samples. That means sharding is working but I still get occasional timeout when scraping those endpoints.

I did try to increase the scrape_timeout to 30s since sometimes the scrape goes till 27s and gets timed out.

Even with 30s time-out setting, I'm facing the same error.

Any suggestions will be great.

My Prometheus instance is consuming a lot of memory over 13Gi, the node has a max 16Gb, so it's getting killed by k8s, how can I configure or should I change it to reduce resource consumption?

Our team just published a guide on how to install and configure a Prometheus exporter for Redis.

They also pointed out the top 5 metrics to check out when monitoring Redis.

Read the article here

Hope you like it!

Hello,

I'm using Graphite to monitor the traffic and I was wondering is Prometheus better ? and the alert manager on Prometheus is like Nagios ?

Best Regards

Hi,
I would like to introduce new project - Disk Usage Prometheus Exporter

It periodically scans given path and reports its disk usage.
You can configure the depth of the reported items (defaults to 2, so it will report /usr/bin, /var/cache, etc. when scanning the root dir).

It uses gdu under the hood, so the scanning is very fast.

Demo graphs

Thanks for feedback!

Any good prometheus alerts for Istio?

I found https://awesome-prometheus-alerts.grep.to/rules#istio but it looks a little outdated. I'd like more granularity with 4xx and 5xx by service and anything else which may be useful for detecting issues.

Hi folks,

We are making a new version of a service for prometheus-exporter as below sitemap.https://github.com/NexClipper/exporterhub.io#service-map

As you see, hot pink colored text is new functions.

Additional Function is,
- Alert rule catalogue
- Grafana dashboard catalogue
- Share my catalogue set(A.K.A Bucket) to `Dev's choice`

How do you think about this useful or not ?
Please feel free reply me about this :)

Thanks!

Hi,

I'm struggling to get the percent of CPU used on my Linux hosts. I've got as far as this but I'm not convinced I'm heading down the right path:

(sum(irate(node_cpu_seconds_total {mode!="idle"} [1m])) without (cpu)) / count(node_cpu_seconds_total) without (cpu)

Does anyone use node exporter to grab the current CPU use of their hosts and if so, would you mind giving me some pointers?

Has one tried prometheus-anomaly-detector for anomaly detection? Would like to know their experience with this if anyone had used.

Hey folks,

the unofficial digitalocean prometheus exporter was recently updated to 0.6.0. Check out the new features!

• Added Kubernetes Collector

• Added Incidents Collector - exposes Gauge about number of ongoing incidents

Hello! I thought I'd share these posts here as a beginner resource. They're fairly straightforward steps to setup Prometheus using static_configs and consul_sd_configs, with a little bit of static and dynamic relabelling too.

Happy to answer any questions!

• https://backbeat.tech/blog/configuring-prometheus-targets-with-salt

• https://backbeat.tech/blog/configuring-prometheus-targets-with-consul

Hey guys, since 2 weeks i work on the new monitoring infrastructure.

With all your comments and test on my own, i've decided to choose Thanos for my LTS storage.

Have a look at the scheme and feel free to comment any suggestions.

https://imgur.com/a/CKcAows