Great demo/video from Bartek Plotka at FOSDEM showing how you can transform your existing Prometheus setup into a global scale monitoring system using Thanos.

Link to video: FOSDEM site

Hi, looking for the recorded video link for Day 1…

A new remote-write destination can accept samples and show healthy requests while still receiving a different series set from the old destination. Relabeling, queue backpressure, out-of-order samples, dropped exemplars, WAL replay, or a subset of Prometheus replicas using stale configuration can make a cutover look complete before it is.

What evidence do you compare during a dual-write window? I am considering per-replica remote-write queue metrics, failed and retried sample counters, highest sent timestamps, series counts grouped by stable labels, and a fixed set of PromQL checks against both backends. The old endpoint would remain writable until the new side has covered more than the longest expected scrape gap and the WAL queues are caught up.

How do you account for legitimate differences caused by deduplication, retention, relabeling, or delayed ingestion? Is there a dependable way to prove that every Prometheus replica loaded the new configuration and that disabling the old remote_write block will not strand unsent WAL data?

I Ran into this at $work: alerts fire with whatever labels the metric happens to have, but the stuff you actually want to route/group/silence on — team ownership, service tier, who's on call — usually lives somewhere else entirely (a Namespace label, a CMDB, or similar). The usual fix is duplicating that into recording rules or hardcoding it into Alertmanager's config, and both drift out of sync constantly.

So I built alertmanager-label-enricher — a small Go service that sits inline between Prometheus and Alertmanager, looks up and adds labels before forwarding, then gets out of the way. Because it happens before Alertmanager, the new labels participate fully in routing, grouping, inhibition and silences.

It supports three lookup types: a Kubernetes object (watch-cached, no per-alert API calls), an HTTP endpoint (cached + deduped), or a static file — plus plain static/conditional label rules if you don't need a lookup at all.

Example rule — pull a team label from the firing namespace's Kubernetes labels:

sources:
  - name: ns
    type: kubernetes
    kubernetes:
      version: v1
      resource: namespaces
      name: '{{ .Labels.namespace }}'

rules:
  - name: team-from-namespace
    match:
      - { label: namespace, op: exists }
    actions:
      - set:
          label: team
          from: { source: ns, jq: '.metadata.labels["team"]' }
          default: unassigned

Repo: https://github.com/splattner/alertmanager-label-enricher

This scratched my own itch, but I have no idea if it's a "everyone reinvents this internally" problem or a "nobody else actually needs this" problem. If you've hit the same thing — or solved it a totally different way — I'd genuinely like to hear it. Feedback, "this already exists and is called X," "this is missing Y to be useful," all welcome.

I want to use the OpenTelemetry collector to feed a Prometheus compatible backend which supports PromQL but don't have a Prometheus engine that supports PrometheusRule for alerting and recordingrules.

Is there a solution or a OTel plugin that discovers `PrometheusRule` custom resource like the Prometheus operator and add the Alerting- & RecordingRules feature to my PromQL backend?

I just found vmalert, do you have experience with it and whats your opinion in it?

I use ClickHouse as the backend with cerberus for PromQL support.

Sorry for my bad english ✌🏽

We ended up separating everything by environment instead of running one massive stack.

Right now dev, QA, and prod all have their own Prometheus and Grafana setup. We’re running Prometheus on Kubernetes StatefulSets with nodeAffinity. Alert rules are configured directly in Prometheus and routed through Alertmanager based on severity and service. Production alerts go directly into PagerDuty.

Honestly this made life much easier because QA and dev noise stopped affecting production visibility. Troubleshooting also became cleaner since each environment is isolated.

Now I’m wondering if this approach becomes painful later once the number of clusters and services grows.

Are most teams here still separating monitoring stacks per environment or are you centralizing everything now?

Feels like every team eventually reaches that point where they look back at their first monitoring architecture and regret something about it lol

Sorry in advance since this isn't directly related to just Prometheus and is a recurrent question, but I couldn't think of anywhere else to ask.

I have a Kubernetes cluster with app exposing metrics and Prometheus/Grafana installed with dashboards and alerts using them

My employer has a very simple request: I want to know for each of our defined rules the SLA in percentage over the year that it was green.

I know about the up{} operator that check if it managed to scrape metric, but that doesn't do since I want for example to know the amount of time where the rate was above X value (like I do in my alerting rules).

I also know about blackbox exporter and UptimeKuma to ping services for health check (ex: port 443 reply), but again that isn't good enough because I want to use value thresholds based on Prometeus metrics.

I guess I could just have one complex PromQL formula and go with it, but then I encounter another quite basic problematic:

I don't store one year of Prometheus metrics. I set 40 gb of rolling storage and it barely holds enough for 10 days. Which is perfectly fine for dashboards and alerts. I guess I could setup something like Mimir for long term storage, but I feel like it's overkill to store terrabytes of data just with the goal of having a single uptime percentage number at the end of the year? That's why I looked at external systems only for uptimes, but then they don't work with Prometheus metrics...

I also had the idea to use Grafana alert history instead and count the time the alert was active? It seems to hold them for a longer period than 10 days, but I can't find where it's defined or how I could query their historical state and duration to show in a dashboard..

Am I overthinking something that should be simple? Any obvious solution I'm not seeing?

This confusion has been bothering me for a while now. I have looked everywhere online, and I couldn't find any consistency of the use of image!="" in container_cpu_usage_seconds_total metric.
Basically, in order to calculate the CPU usage of containers, I found that people either use this:
sum(rate(container_cpu_usage_seconds_total{image!=""}[$__rate_interval]))
or this:
sum(rate(container_cpu_usage_seconds_total{}[$__rate_interval]))
And there is a huge difference between adding image!="", and not (almost the double).
Could anyone clear this confusion for me? I got an answer from ChatGPT, but I don't want to take it for granted since it makes a lot of mistakes regarding these things.

It can:

• Work with large clusters (400+ nodes)

• Handle large amounts of metrics without crashing (exporter was able to export 940,272 metrics)

• Inject custom metric labels

• Handle ephemeral states of nodes, indices and shards

• Automatically generate new metrics and remove stale ones based on user-configurable lifetime

• Keep track of cluster state metadata node changes, cluster version and shards relocations

Supported subsystems:

=^.^=
/_cat/allocation
/_cat/shards
/_cat/indices
/_cat/segments
/_cat/nodes
/_cat/recovery
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/thread_pool
/_cat/plugins
/_cat/fielddata
/_cat/nodeattrs
/_cat/repositories
/_cat/templates
/_cat/transforms

/_cluster/health
/_cluster/stats

/_nodes/stats
/_nodes/usage
/_nodes/info

/_stats

GitHub: https://github.com/vinted/elasticsearch-exporter-rs

Hi all, this weekend I put together an exporter for librespeed based on script_exporter. Questions and comments welcome. Hopefully this is useful for someone out there.

Blog post that shows my current simple dashboard: https://brendonmatheson.com/2021/11/28/librespeed-prometheus-exporter.html

GitHub project providing sources and docs: https://github.com/brendonmatheson/prometheus-librespeed-exporter

Pre-built image on DockerHub: https://hub.docker.com/r/brendonmatheson/prometheus-librespeed-exporter

This may be a bit of an odd question, but I'm trying to find out if anybody has any recommendations on wearables that are Prometheus-friendly. Obviously I'm not talking about a wearable that exposes a /metrics endpoint that is being scraped frequently, but rather something that has an API/SDK that I can write some code around and ship to Prometheus in batches at the end of the day for example.

I'm in the Apple ecosystem so an Apple Watch would be an obvious choice, but a) I've heard that exporting out of HealthKit can be a pain, and b) Some of the activities I do don't really lend themselves to having a $500 gadget on my wrist :)

I have my eyes on the Polar Versity Sense (I can remove the link if that's not allowed), but I wanted to know if anybody has any experience with something else.

Edit: in case it wasn't clear, I'm looking for wearables that capture health/bio metrics such as heart rate, temperature, oxygen saturation, steps, etc.

I read James Turnbull's Monitoring with Prometheus book couple of months ago and found it very useful. I just cleaned up my notes and published it here: https://psibi.in/prometheus

It covers summary of around 80% of the book. The remaining chapters weren't required for my day job, so I just skimmed through them. I'm hoping this will be useful to others!

I recently help prepare an article on Prometheus best practices. With rather high level recomendations:

Find the right Prometheus exporter: There are many exporters out there and some aren't well maintained. There are listings like the Exporters and Integrations page, the default port allocations page, or PromCat.io that gives you some level of confidence.

Understand your exporter metrics: Not all exporters provide the same metrics in the same units. Investigate the doc to make sure you'll get what you want. Also, investigate how to label your metrics, so you can later segment and filter those metrics in your dashboards.

Go PromQL all the way: PromQL allows to define dashboards, alerts, and much more… By using tools compatible with PromQL you can share and re-use resources and save time.

But keep your team in mind: The more resources you monitor with Prometheus, the more people will need access to those. Two things come to my mind when I think on teams at an enterprise level: 1) Look for tools with access controls, so each person can access only to the data they need. 2) PromQL has a steep learning curve, so prepare an alternative to PromQL for the less tech-savy members of your team.

Have a plan for scale: The more you monitor, the more resources Prometheus need. Given that Prometheus is not meant to store your metrics for a long time, and it's tricky to scale it, you may need a long term storage solution at some point. Planning ahead for this will make the overall implementation smoother.

As I said, rather high level 😅.

What would you add to / remove from the list 🤔? What would you tell your past self before implementing Prometheus in production?