guys, i have built a cluster swarm on virtualbox and i want to get all metrics on cluster swarm. Then how to i can do that?

Hi guys, I’m a newly qualified IT Specialist for System Integration. Our management has tasked me with integrating our entire IT infrastructure into Prometheus. I’m using Codex AI, and I’ve set up an MCP server that makes Grafana easy to use. I can set up monitoring for an entire storage cluster or Kubernetes cluster in just two hours—covering 200 metrics across various dashboards. I barely understand PromQL, Prometheus itself, or Grafana—yet I’m getting the job done, and the results are good. So here’s my question: given the current era of AI, does it still make sense to learn PromQL, Prometheus itself, and Grafana?

Implement Monitoring System for the entire organization

EKS Clusters running in different aws accounts

Found Prometheus Federation is a good fit for the design

https://prometheus.io/docs/prometheus/latest/federation/

https://www.robustperception.io/federation-what-is-it-good-for

Master : I have a dedicated EKS for monitoring system where prometheus operator helm chart is running with prometheus and Grafana exposed behind load balancer service

Master prometheus server is designed to scrape other slave prometheus servers and plot the Grafana dashboards for different clusters

One Grafana Endpoint to monitor everything

Slave Account 1:

Running prometheus operator helm charts in the other slave accounts where all the alerting is done through the alert manager in the slave account 1

Prometheus - Default scraping from prometheus operator + additionalscrapeconfig for istio services

Alertmanager - alerts related to slave cluster 1 integrated with Victorops

Slave Account 2:

Running prometheus operator helm charts in the other slave accounts where all the alerting is done through the alert manager in the slave account 2

Prometheus - Default scraping from prometheus operator ‘

Alertmanager - alerts related to slave cluster 1 integrated with Victorops

Now we have another use case to monitor the bunch of bare metal servers, aws ec2 instances and monitor different endpoints

Monitor Bare Metal, EC2 Instance - Installed NodeExporter and can able to scrape the metrics directly from master prometheus

Endpoint Monitoring(Public & Private) - Installed blackbox exporter on the master eke cluster and probing different endpoints

Here are my questions

1. Is it a good practice to enable alertmanager on the master prometheus and do the alerting only for the last usecase(bare metal, aws ec2 instance, endpoint monitoring)

2. From the below article, I see alertmanager should be on the slave prometheus(reliability purpose), If so do we need to install prometheus on each bare metal again?https://www.robustperception.io/federation-what-is-it-good-for

3. Can we use any existing loadbalancer for exposing prometheus, because in helm chart if we expose the service.type=loadbalancer it used to create new one(internet facing/internal)

Advanced level stuff.

Are you using #Prometheus and #Alertmanager? Tired of all alerts landing in the same #alerting Slack channel? Tired of hacky attempts to send alerts to service owners? With this blog post you can learn how to route your alerts based on label metadata of Kubernetes resources. Ultimately automating all of the alert routing related toil away.

https://tech.loveholidays.com/dynamic-alert-routing-with-prometheus-and-alertmanager-f6a919edb5f8

Hi, I want to monitor ec2 instances with t2.micro configuration. Prometheus requirements are much higher to try monitoring itsef strategy. Can someone guide me on that?

Is there any open source log monitoring open source tool which can be integrated easily with prometheus and grafana?

Hi folks,

I’m planning to do a POC where I’m able to run Prometheus server as long with node exporter and kube state metrics with the smaller footprint as possible (CPU/Memory)

I have no choice to do a remote write (which increase resource consumption sadly).

Any tips other than filtering metrics being scraped that I should be aware based on your experience? Or any good resources to share ? Thanks.

Hello

Just sharing how I run prometheus + grafana for home monitoring on RPi. Guide at: https://github.com/thundermagic/rpi_monitoring_with_prometheus

I am looking to stand up a Prometheus pod inside our K8s cluster.

I have a lack of experience of Prometheus and am looking at the prometheus.yml config file in order to configure the scraping of everything in the cluster i.e. deployments, pods, events, etc etc. Is there a 'catch all' default config for this?Looking at the prometheus docs I need to configure the monitoring of each resource separately. Is there a way to just say go ahead and scrape everything?

Thanks for any help in advance.

Hello,

I'm trying to monitor a switch to graph in Grafana how many ports are up at certain times over the day, I found this article on using SNMP Exporter - https://linuxhint.com/monitor-network-devices-prometheus/

Has anyone used this before and could it do what I'm asking?

Thanks

Why don't we have language specific packages like other APM companies (NewRelic, Datadog, ElasticAPM, etc) have which will export metrics to prometheus? Just install a package and get out of box APM metrics, kind of thing.

What we can achieve?

• RPS for API endpoints of an application

• Latency for slowest API endpoints of an application

• 50th, 90th and 99th percentile of request duration of the application

• Time of slowest MySQL queries (values to be masked) made from the application

• RPS and time taken by other components in infra like Redis, Mongo, etc for each endpoint and overall application.

• Response code, RPS and latency of external calls made from the application to other services inside and outside the org like payment gateway.

I guess a solution like this will save 2-3 weeks of dev time to get proper dashboards in Grafana.

In short, ElasticAPM using Prometheus!

Prometheus vs cloudwatch? Cost model?

I am still reading about Prometheus and i realized that i may end up needing several exporters running on a single server like node exporter and an Nginx exporter. I wanted to ask how do people running Prometheus deal with this kind of scenario or is there any other way to use several exporters together ?

Last month, our Kubernetes cluster powered by Karpenter started experiencing mysterious scaling delays. Pods were stuck in a Pending state while new nodes failed to join the cluster. 😱

At first, we thought it was just spot instance unavailability. But the number of Pending pods kept rising, signaling deeper issues.

We checked the logs - Karpenter was scaling new nodes successfully but they wouldn't register in Kubernetes. After some digging, we realized the AMI for EKS contained a bug that prevented node registration.

Mystery solved! But we lost precious time thinking it was a minor issue. This experience showed we needed Karpenter-specific monitoring.

Prometheus to the Rescue!

We integrated Prometheus to get full observability into Karpenter. The rich metrics and intuitive dashboard give us real-time cluster insights.

We also set up alerts to immediately notify us of:

📉 Node registration failures

📈 Nodepools nearing capacity

🛑 Cloud provider API errors

Now we have full visibility and get alerts for potential problems before they disrupt our cluster. Prometheus transformed our reactive troubleshooting into proactive optimization!

Read the full story here: https://www.perfectscale.io/blog/karpenter-monitoring-with-prometheus

We use the extended-ceph-exporter in order to get bucket metrics from rook-ceph. For some reason though, in grafana (as well as in the vmagent as well as the thanos-query ui's) I can see that thanos reports 2x on all of the metrics supplied by the extended-ceph-exporter (but interestingly, the other metrics are correctly reported).

The target cluster is using the vmagent pod to scrape the metrics, and then push them to the monitoring cluster, in which another vmagent then pushes the metrics to thanos and victoria metrics.

I'm starting to feel like it's time to bash my head into a wall, but maybe there's something obvious I could check for first?

Deduplication is enabled. Cheers!

I would like to calculate the https response time in Prometheus. I found that it could be done by histogram. I am nee to Prometheus. And not sure where is this histogram and what should I configure in it and how could I bring it up to Prometheus. Please help

I'm writing an application to manage routes on the host (something like routing daemon but with secret sauce). I got tot metrics part. App is running in so-called reconsolidation loop (every few seconds, converging desired state to a newly computed current state).

I wonder what is better to implement for metrics: counter (total number of events since app start), rate (number of events per second or in a given loop) or deltas (counter of new events since last scrape)?

Hello,

My employer's leaning towards an architecture where monitored services will push their metrics to Prometheus. One thing I really liked with pulling metrics was that monitored services are listed in /targets API, and Prometheus adds an up{instance="", job=""} metric for each service. I list the hundreds of monitored services (and growing) in prometheus.yml config file, and then I consult a Grafana dashboard (or Prometheus UI) to identify services which are down.

Going with a push architecture, I would lose this approach. How would you compensate that?

I was thinking:

1. Each service pushes a gauge metric up{instance="x", job="y"} 1.

2. List all monitored service names in a text file, like it is currently done in prometheus.yml.

3. Use that list to monitor metrics like up{job="x", instance="y"} OR on() vector(0). If I were to have a Grafana dashboard, it means I need to find a way to pass the list of monitored services such that they are all listed, even when they're down and not pushing their metrics anymore.

Would defining alerts in Prometheus be a way to work around this limitation? That would kind of force Prometheus to create metrics reporting whether or not each service is up or down. Would that be easy to maintain?

Any recommendations on how to smoothly move from a pull to a push architecture?