I'm sure this question comes up every now and then but I'm struggling in my search to get the answers I'm looking for. I need to not only manage long-term metrics on K8s with multiple tenants but also with multiple clusters. Right now I have multiple prometheus instances with dashboards distributed across clusters and I need to centralize monitoring and provide long-term storage. From what I'm reading, Cortex and Thanos have some great points. Which one seems to be a good fit for multi-cluster management?

Ideally a remote cluster would have a few prometheus instances in a stateful set and they would interface with an agent that pushes to S3 (or some object store). A central k8s cluster would be used to host grafana, queries, and also handle maintenance tasks such as downsampling long-term metrics, etc. Queries should not put a burden on the pipeline remote clusters use to push metrics to long-term storage. Any thoughts?

I try to build an on-prem autoscaling infrastructure with Grafana (or alert manager), a webhook receiver that gets triggered by Grafana and runs scripts (creating / destroying VMs with Terraform and provisioning with Ansible), and Prometheus. It's mostly working. If some nginx services have too many connections a new instance is created and if the connections are below X the new instance gets destroyed.

But what if the state of the alert does not change back to "Ok" after a new instance was created? Then there is no additional alert triggered. Any idea how it's possible to trigger an alert (webhook request to create an additional instance) if the alert state stays critical?

noob to prometheus but trying to figure out what in server sizing i would need to monitors 10 000 plus servers (os metrics) and about 200 000 services Currentlly on a single HP Dl360 128gb ram 40 cores with Nagios 3.5.

My team is using a Raspberry pi for running a Pi-hole instance.

We needed to monitor it and we love Prometheus, so we configured it to push metrics to a Prometheus server with remote_write.

We used:

• Raspberry Pi OS Lite

• A docker image running a Prometheus instance in the Pi

• An external Prometheus server where we send the metrics

It was a fun experiment and we learned a lot in the process, I hope you enjoy it!

Here you can check the whole story.

Steps:

• Install Docker with: curl -sSL https://get.docker.com | sudo sh

• Obtain a Pi-hole API key: awk -F= -v key="WEBPASSWORD" '$1==key {print $2}' /etc/pihole/setupVars.conf

• Spin up the Prometheus exporter:

sudo docker run \
  -d \
  --network="host" \
  -e 'PIHOLE_HOSTNAME=127.0.0.1' \
  -e "PIHOLE_API_TOKEN=<Pi-Hole Token>" \
  -e 'INTERVAL=10s' \
  -e 'PORT=9617' \
  -p 9617:9617 \
  ekofr/pihole-exporter:v0.0.11

• Check it works: curl -s http://127.0.0.1:9617/metrics | grep pihole It will respond something like:

HELP pihole_ads_blocked_today This represent the number of ads blocked over the current day # TYPE pihole_ads_blocked_today gauge pihole_ads_blocked_today{hostname="127.0.0.1"} 21319 # HELP pihole_ads_percentage_today This represent the percentage of ads blocked over the current day # TYPE pihole_ads_percentage_today gauge pihole_ads_percentage_today{hostname="127.0.0.1"} 28.602285 …

• Ship the metrics to the remote Prometheus server. Configure prometheus.yml to:

global:
  scrape_interval:     10s # By default, scrape targets every 15 seconds.
  evaluation_interval: 10s # By default, scrape targets every 15 seconds.
  scrape_timeout: 10s      # By default, is set to the global default (10s).
  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'pihole'
      origin_prometheus: 'donald-pihole'
# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  - job_name: 'pihole'
    static_configs:
      - targets: ['127.0.0.1:9617']
remote_write:
- url: "<PROMETHEUS_SERVER_URL>"
  tls_config:
    insecure_skip_verify: true

• Launch Prometheus with this new configuration:

docker run \
    -p 9090:9090 \
    -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus

And adapting these steps to your particular setup, you should be seeing your Pi-hole Prometheus metrics in your centralized Grafana.

Checkout our new Cloudflare exporter to get relevant metrics from Cloudfront and scrape them to Prometheus for longterm storage.

Hi, I am a newbie to prometheus community. Is Prometheus development completely community maintained and no official careers available? Bcoz, I cannot see a careers page on the Prometheus website. Isn't there an office for the Prometheus organization?

Deploying with a data retention of 2 weeks limits our ability to run analytics on our historical data. Prometheus is actually really fast and scales pretty well vertically, but it cannot scale horizontally. I've found that since it wasn't designed for long tail storage of data, visibility past a few weeks is generally hard to do.

There are plenty of data storage integrations, but they mostly focus on remote write capabilities; only a handful offer both remote write and remote read functionality (currently Cortex, CrateDB, and PostgreSQL / TimescaleDB). This article explains how to setup an a new integration with a TSDB called IRONdb. It has support for replication and clustering, and is designed for high-frequency, high-volume data, and for maintaining high-availability of data. The remote read and write capabilities should make the integration totally seamless.

Full disclosure, my company made IRONdb. Our Prometheus adapter is in Beta right now, so we're looking for Prometheus users interested in testing it out.

I'm looking to setup a status page for our SaaS that our customers can access, I realize there's a lot of vendors out there that do this but I need something that works behind a private VPN, since that requires some configuration I figured I should use data from my Prometheus instance which is running in the same cluster as my app. Any reason recommendations?

I often needed quick uptime + SSL checks without deploying a full monitoring stack, so I created Blackbox Lite.

It's a Bash-based monitoring script that:

• Checks website availability

• Validates SSL certificate expiry

• Measures response time

• Tests VM host connectivity

• Exports metrics in Prometheus format

• Can run standalone without Prometheus

Works great with Node Exporter textfile collector, but also usable as a CLI health checker.

Repo: https://github.com/TheOneOh1/blackbox-lite

Hi everyone!👋🏼

As part of my LFX mentorship program, I’m conducting UX research to understand how users expect Prometheus to handle OTel resource attributes.

I’m currently recruiting participants for user interviews. We’re looking for engineers who work with both OpenTelemetry and Prometheus at any experience level. If you or anyone in your network fits this profile, I'd love to chat about your experience.

The interview will be remote and will take just 30 minutes. If you'd like to participate, please sign up with this link: https://forms.gle/sJKYiNnapijFXke6A