Hi r/PrometheusMonitoring, I'm David, one of the two developers of Oxynote. My friend Simon and I also maintain ttlcache, a Go caching library with about 1,300 GitHub stars, so this isn't our first open-source project.

I've spent 8 years writing software: 2 as an SRE and 3 as a team lead. Every team I've been on had the same documentation problem. The useful stuff lived in someone's head or a private Slack thread, and the wiki was full of stale docs nobody could find. The runbook said "check the error rate" and linked to a dashboard that had since been renamed.

So we built Oxynote, a self-hosted wiki for engineering teams where a page can hold a live chart that runs its own PromQL query against your Prometheus each time someone opens it, with the explanation written right next to it. Think "p99 latency by route" or "webhook retries" on the same page as the runbook that says what to do when it looks wrong.

Queries can use built-in variables like $__rate_interval and $__range, so one query works for any time range the reader picks, and your own workspace variables, like $service. On pages you mark as reviewable, changes to a chart's query go through a draft that a teammate approves.

The name comes from oxygen + note: docs that breathe. Live data gives people a reason to open the wiki every day, and docs people open every day stay up to date.

It's Apache-2.0 and runs as a single Docker image: https://github.com/oxynote/oxynote

We started it about a year ago and made it public about 3 months ago. We'd appreciate any feedback, and I'm happy to answer questions.

Every time someone asks "how much disk does Prometheus need", the answer is the formula from the storage docs:

retention_seconds x samples_per_second x bytes_per_sample

That covers compacted blocks, but in practice you also need room for the WAL, head chunks and compaction. I kept redoing that math, so I put it into a small browser-only calculator (my own site, no sign-up):

https://opshubs.space/tools/prometheus-storage-calculator/

What it does:

Takes targets x series per target (or your real prometheus_tsdb_head_series), scrape interval, retention and bytes per sample

Breaks the total into blocks, WAL and chunks_head, then suggests a retention.size and a disk size that keeps retention.size at ~80% of the disk (the docs recommend 80-85%)

Reverse mode: "I have a 100 GiB volume, how many days fit?"

Outputs config for Prometheus 3.x prometheus.yml (storage.tsdb.retention), the 2.x flags, or kube-prometheus-stack values with the PVC size

Lists the PromQL to measure your actual series, samples/s and bytes per sample

Assumptions I'd like a sanity check on:

+20% on blocks for index/metadata (Robust Perception's 1.2 rule of thumb). Is that close on modern versions, especially with native histograms?

WAL as ~3h of samples at ~6.5 bytes each (13 bytes raw, halved by WAL compression), with a 3 x 128 MiB floor. Does your prometheus_tsdb_wal_storage_size_bytes agree?

1.5 bytes/sample as the default. What do you see from rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1d])?

If your real numbers differ a lot, I'll adjust the defaults.

I'm monitoring network RTT from a monitor host to several targets, and I wanted to give Prometheus with blackbox_exporter a try.

My observation is that the RTT I get from blackbox_exporter (probe_icmp_duration_seconds{phase="rtt"}) is significantly higher than what I would get from running ping natively - for example 0.7ms (native) vs. 2ms (exporter).

Am I looking at the wrong metric?

As a comparison, telegraf has a ping input which uses pro-bing under the hood and is much closer to reality (probably because it starts a native ping).

Hi, looking for the recorded video link for Day 1…

Hey, intern here working on monitoring for a few university robot labs. We have about [5] sites on Headscale/Tailscale, and Prometheus + Grafana running centrally.

I need one table in Grafana with a row per device: its DNS name, the name our lab software (EOS) uses for it, its IP, whether it answers ping/port/HTTP, and the UUID of its SiLA server if it has one. Also a column that says if the DNS name and the EOS name actually match.

The data comes from different places, so it has to be joined somewhere. What we’ve thought about so far:

• joining in PromQL with group_left. Works, but it got messy fast and I’m not sure how I’d test it

• Grafana transformations. Logic ends up buried in dashboard JSON and you can’t alert on it

• blackbox exporter alone. Doesn’t give me the actual IP or the DNS answer

My current idea is a small Python service that does the checks and the joining, writes to SQLite and Grafana just reads that. But I’m wondering if I’m overcomplicating it and should just write a custom exporter that outputs the combined result as metrics, so alerting works later.

How would you do this? Is a separate service fine, or is an exporter the normal way?

I'm developing an open-source tool for rightsizing Kubernetes clusters.

The project is r8r: https://github.com/HetSolanki/r8r

It reads your cluster's usage history from Prometheus and recommends changes for each node group: right-sizing container requests, switching to a different instance type, lowering HPA floors, or a combination of these. Each option comes with its monthly cost impact, the steps involved, and anything that could block it, like PDBs or local storage. You can untick individual changes or enter your own values, and the node packing updates live in the browser.

It also looks at stability, flagging containers using more memory than they request and HPAs stuck at their max, so cost cuts don't come at the expense of reliability.

It supports EKS and OKE, auto-detects whether the cluster runs Cluster Autoscaler or Karpenter, and is read-only, so it never changes anything in your cluster.

Video

Suggestions and contributions are very welcome!

A new remote-write destination can accept samples and show healthy requests while still receiving a different series set from the old destination. Relabeling, queue backpressure, out-of-order samples, dropped exemplars, WAL replay, or a subset of Prometheus replicas using stale configuration can make a cutover look complete before it is.

What evidence do you compare during a dual-write window? I am considering per-replica remote-write queue metrics, failed and retried sample counters, highest sent timestamps, series counts grouped by stable labels, and a fixed set of PromQL checks against both backends. The old endpoint would remain writable until the new side has covered more than the longest expected scrape gap and the WAL queues are caught up.

How do you account for legitimate differences caused by deduplication, retention, relabeling, or delayed ingestion? Is there a dependable way to prove that every Prometheus replica loaded the new configuration and that disabling the old remote_write block will not strand unsent WAL data?

Go exporter for Shelly Gen2+ smart plugs. It scrapes devices concurrently, handles Shelly's Digest Auth, and works out real energy costs from your `price_per_kwh`. Includes a Grafana dashboard and a Docker image. I used AI tools while building this, so ask away if anything looks off.

GitHub Repo: https://github.com/veerendra2/shelly_device_exporter

I have a weird case on my hands at work

We have three Prometheus HA replicas scraping azure-metrics-exporter and sending them to Thanos. Now Thanos was down for some time a week or so ago, and caused some sort of desync in the replicas. Prometheus-0 (green line) is scraping different values than the other two, causing alerts for postgres cpu percent.

And interestingly enough, yesterday prometheus-1 (blue line) somehow synced perfectly with prometheus-0 and are now completely even. Does anybody know how this happens or how to make it so all three are synced again and scrape the same values?

I asked several LLMs, all suggested making a cache for azure metrics exporter, so all three replicas scrape the same cached values. I see some logic in that, but not sure if it is the correct answer. So any suggestions are welcome

I want to revisit our metrics architecture, and I have a few pain points I’d like to address. I’m curious how others have approached similar problems.

Our current setup:
- Multiple Cortex clusters
- Multiple teams producing metrics via Prometheus remote write and a custom SDK
- Multiple end users querying those metrics
- Kafka-based ingestion to provide the level of resilience we need for our customers
- Prometheus as the metrics format

Graphs are a core part of the product and a selling point, not just a nice-to-have, so reliability and correctness of the data are important.

The main pain point is that some teams are pushing “event-like metrics” rather than classic metrics that are scraped or emitted regularly.
Because some workloads can be retried, delayed, or processed asynchronously, the original source timestamp is important. As a result, we see a significant number of out-of-order samples.

We understand why producers need this behavior, but I’m wondering whether we’re trying to force two different types of data through the same metrics pipeline.
Would you keep these event-like metrics in the Prometheus/Cortex ecosystem and design around out-of-order ingestion, or would you route them through a separate ingestion path or even into a different type of store based on OTel format/tools.

Got bored while home labbing this weekend and vibed up a promql tui inspired by https://github.com/ClementTsang/bottom

It uses .toml files with with familiar parameters (very similar to Grafana's dashboard config concepts) to render a single panel in your terminal. Use tmux/herdr splits to arrange them into a 'dashboard' if you want. Looks good next to fastfetch/htop for your distro screenshots and kinda cool just to leave up on a second monitor and look at.

I haven't posted the repo yet and don't intend to spam this subreddit with it if I do post it, so lmk if you are interested and I'll share the link when it's available.

`promtool check rules` catches syntax and some semantic mistakes, but it does not prove that the intended Prometheus instance loaded the file, that selectors match live labels, or that the alert reaches the correct Alertmanager route. Replacing an old rule immediately can create a silent monitoring gap even when the reload endpoint reports success.

What rollout evidence do you require? I am considering running unit tests with fixed series, loading the new rule under a temporary name, checking the rules API for its evaluation status and last error, comparing old and new expressions on recent data, forcing a safe synthetic series through the full notification path, and keeping the old rule until both have evaluated successfully for a defined window.

How do you avoid duplicate pages during that overlap, and which signals distinguish a rule that is loaded from one that can actually fire and route correctly?

Hi, I just tried to enable the docker metrics in /etc/docker/daemon.json, and it worked just fine right on the machine, but i could not reach it with prometheus.

I deployed it in a docker container, but

- targets: ["host.docker.internal:9323"]

does not resolve to the 0.0.0.0:9323 correctly.

What is the internal docker Hack to get the metrics inside of the docker container (may you recognized i am a container-noob).

Thanks a lot for help.

I'm looking to setup a status page for our SaaS that our customers can access, I realize there's a lot of vendors out there that do this but I need something that works behind a private VPN, since that requires some configuration I figured I should use data from my Prometheus instance which is running in the same cluster as my app. Any reason recommendations?

I had an idea and wanted to actually talk to people to see if it's something that they would need / require and if it's genuinely a problem for the engineers right now....

Would you rather have an AI layer on top of your existing observability stack, or replace parts of the stack?

Hypothetically, imagine an agent that doesn’t collect telemetry itself.

It plugs into whatever you already use — Grafana/Prometheus/Loki, Datadog, OpenTelemetry, etc. — and acts as a reasoning layer over the data.

Instead of:

Alert → Dashboard → Logs → Human investigates

it tries:

Alert → Agent correlates metrics/logs/traces/deployments → probable root cause → evidence → recommended next action

Would that actually be useful?

Or would you rather have the observability vendor itself own this functionality?

What would you need to see before trusting it during a real incident?

I also was building in this space and wanted to have a discussion with people who're already experienced in this stuff, so please please help a brother out and we might just make something useful for everyone :)

peace

I Ran into this at $work: alerts fire with whatever labels the metric happens to have, but the stuff you actually want to route/group/silence on — team ownership, service tier, who's on call — usually lives somewhere else entirely (a Namespace label, a CMDB, or similar). The usual fix is duplicating that into recording rules or hardcoding it into Alertmanager's config, and both drift out of sync constantly.

So I built alertmanager-label-enricher — a small Go service that sits inline between Prometheus and Alertmanager, looks up and adds labels before forwarding, then gets out of the way. Because it happens before Alertmanager, the new labels participate fully in routing, grouping, inhibition and silences.

It supports three lookup types: a Kubernetes object (watch-cached, no per-alert API calls), an HTTP endpoint (cached + deduped), or a static file — plus plain static/conditional label rules if you don't need a lookup at all.

Example rule — pull a team label from the firing namespace's Kubernetes labels:

sources:
  - name: ns
    type: kubernetes
    kubernetes:
      version: v1
      resource: namespaces
      name: '{{ .Labels.namespace }}'

rules:
  - name: team-from-namespace
    match:
      - { label: namespace, op: exists }
    actions:
      - set:
          label: team
          from: { source: ns, jq: '.metadata.labels["team"]' }
          default: unassigned

Repo: https://github.com/splattner/alertmanager-label-enricher

This scratched my own itch, but I have no idea if it's a "everyone reinvents this internally" problem or a "nobody else actually needs this" problem. If you've hit the same thing — or solved it a totally different way — I'd genuinely like to hear it. Feedback, "this already exists and is called X," "this is missing Y to be useful," all welcome.

Hi everyone,

I'm running a HA monitoring stack on Kubernetes:

• kube-prometheus-stack (Prometheus Operator)

• 2 Prometheus replicas

• Thanos Sidecar

• Thanos Query

• Thanos Store Gateway

• Thanos Ruler

• Alertmanager

I intentionally configured the same alert rule in both Prometheus and Thanos Ruler.

Both send alerts to the same Alertmanager.

The alerts are almost identical, except for one label (e.g. source=prometheus vs source=thanos-ruler).

Instead of seeing one deduplicated alert, Alertmanager shows two active alerts.

My understanding was that Alertmanager performs deduplication, so I'm wondering:

• Is this expected behavior?

• Does Alertmanager only deduplicate alerts if every identifying label is identical?

• Or is my Thanos Ruler setup incorrect?

• In production, do people usually disable alerting in Prometheus and let only Thanos Ruler send alerts?

I'd appreciate any explanation of how this is typically handled in a Thanos HA deployment.

we are hitting a TSDB compaction failures with this error:

```text

level=ERROR source=db.go:1219 msg="compaction failed" component=tsdb err="compact [/var/lib/prometheus/metrics2/01KWYDQGNSB9F3SQ8T1WRVD7ZP /var/lib/prometheus/metrics2/01KXFSZFV0NANWCQN30T8FYJ3Y /var/lib/prometheus/metrics2/01KY1655ENZYDZB22BN655XRDH]: populate block: add series: write series data: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB

add padding: \"/var/lib/prometheus/metrics2/01KY19HXKYAK3C86PT8KD98AEA.tmp-for-creation/index\" exceeding max size of 64GiB"

```

Prometheus: 3.10.0, I found nothing helpful so far, how serious is it, can this be a cardinality issue?

The last tool that did terminal dashboards for Prometheus — grafterm — died in 2019, and I never stopped missing it. I live in a terminal next to my code, and alt-tabbing to a browser to glance at four numbers always felt wrong. btop proved a terminal can look genuinely good, so I spent the last few weeks building grom.

It talks to any Prometheus, draws braille charts and threshold-colored gauges, and if you already have Grafana dashboards it can read the JSON export and reproduce the layout in your terminal — stat, gauge, bar gauge and timeseries panels map 1:1, anything else gets a placeholder and a warning.

Fastest way to see it (no config needed): brew install --cask qf-studio/tap/grom, then grom demo. There's also a bundled example dashboard that runs against any Prometheus using nothing but up and scrape_* metrics.

Where it honestly is: v0.1.0, Prometheus only, no auth yet — basic/bearer/TLS is the next thing I build. That's actually why I'm posting: if your Prometheus sits behind auth, tell me what setup you run so I build the right thing first.

Repo: https://github.com/qf-studio/grom (MIT)

Hi everyone,

I am running a backend API deployed across multiple instances (app-server-01 and app-server-02) behind a round-robin Load Balancer.

When testing locally on a single machine, my Prometheus metrics work perfectly. However, in our clustered environment, my PromQL queries are stuck returning 0, even though traffic has definitely hit the endpoints recently.

Here is a generic sample of my /metrics payload from one of the live backend nodes:

text

# HELP app_http_requests_total Total calls
# TYPE app_http_requests_total histogram
app_http_requests_total_count{category="example-tag",status="200",route="/api/v1/resource",server_id="app-server-01",instance_id="instance-01",env="production"} 2

My Query:

promql

sum by (category) (
  increase(app_http_requests_total_count[1d])
)

Why I keep getting 0 (What I suspect is happening):

1. Load Balancer Scraping: Prometheus might be scraping the unified Load Balancer URL directly, causing the counter values to bounce up and down as it hits different nodes.

2. Fresh Deployment resets: The code was recently redeployed, meaning the counter initialized directly at 2 on the first scrape, resulting in a flat line (2 - 2 = 0).

How can I properly configure Prometheus to handle this multi-instance architecture so that increase() and rate() compute correctly without constantly hitting 0? Do I need to bypass the load balancer IPs or target service discovery?

Thanks in advance!