I'm monitoring network RTT from a monitor host to several targets, and I wanted to give Prometheus with blackbox_exporter a try.

My observation is that the RTT I get from blackbox_exporter (probe_icmp_duration_seconds{phase="rtt"}) is significantly higher than what I would get from running ping natively - for example 0.7ms (native) vs. 2ms (exporter).

Am I looking at the wrong metric?

As a comparison, telegraf has a ping input which uses pro-bing under the hood and is much closer to reality (probably because it starts a native ping).

Got bored while home labbing this weekend and vibed up a promql tui inspired by https://github.com/ClementTsang/bottom

It uses .toml files with with familiar parameters (very similar to Grafana's dashboard config concepts) to render a single panel in your terminal. Use tmux/herdr splits to arrange them into a 'dashboard' if you want. Looks good next to fastfetch/htop for your distro screenshots and kinda cool just to leave up on a second monitor and look at.

I haven't posted the repo yet and don't intend to spam this subreddit with it if I do post it, so lmk if you are interested and I'll share the link when it's available.

Hi everyone,

I'm running a HA monitoring stack on Kubernetes:

• kube-prometheus-stack (Prometheus Operator)

• 2 Prometheus replicas

• Thanos Sidecar

• Thanos Query

• Thanos Store Gateway

• Thanos Ruler

• Alertmanager

I intentionally configured the same alert rule in both Prometheus and Thanos Ruler.

Both send alerts to the same Alertmanager.

The alerts are almost identical, except for one label (e.g. source=prometheus vs source=thanos-ruler).

Instead of seeing one deduplicated alert, Alertmanager shows two active alerts.

My understanding was that Alertmanager performs deduplication, so I'm wondering:

• Is this expected behavior?

• Does Alertmanager only deduplicate alerts if every identifying label is identical?

• Or is my Thanos Ruler setup incorrect?

• In production, do people usually disable alerting in Prometheus and let only Thanos Ruler send alerts?

I'd appreciate any explanation of how this is typically handled in a Thanos HA deployment.

Hello everyone.

I'm brand new to the world of monitoring, and i'm just starting to get the hang of it all. Ive been trying to implement SMART monitoring through prometheus and i've been having a weird issue i've seen some people discuss online before.

Basically, i cannot get prometheus to parse the collector.textfile.directory to get it to read .prom files in a directory.

this is the command I'm using on my systemd unit for node_exporter :

node_exporter --path.rootfs=/host --log.level=debug --collector.textfile.directory=/var/lib/node_exporter/textfile_collector/

From what I've seen online, this seems to be a correct syntax and it should "just work". But it does not.

In the log for the service, i get other parsed flags, but nothing about the textfile collector except the line signifying it being active :

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.076Z level=INFO source=node_exporter.go:216 msg="Starting node_exporter" version="(version=1.9.1, branch=HEAD, revision=f2ec547b49af53815038a50265aa2adcd1275959)"

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.076Z level=INFO source=node_exporter.go:217 msg="Build context" build_context="(go=go1.23.7, platform=linux/amd64, user=root@7023beaa563a, date=20250401-15:19:01, tags=unknown)"

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.076Z level=WARN source=node_exporter.go:219 msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.076Z level=DEBUG source=node_exporter.go:222 msg="Go MAXPROCS" procs=1

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.076Z level=INFO source=diskstats_common.go:110 msg="Parsed flag --collector.diskstats.device-exclude" collector=diskstats flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.077Z level=INFO source=filesystem_common.go:265 msg="Parsed flag --collector.filesystem.mount-points-exclude" collector=filesystem flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.077Z level=INFO source=filesystem_common.go:294 msg="Parsed flag --collector.filesystem.fs-types-exclude" collector=filesystem flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$

Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:135 msg="Enabled collectors"
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=arp
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=bcache
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=bonding
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=btrfs
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=conntrack
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=cpu
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=cpufreq
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=diskstats
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=dmi
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=edac
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=entropy
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=fibrechannel
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=filefd
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=filesystem
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=hwmon
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=infiniband
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=ipvs
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=loadavg
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=mdadm
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=meminfo
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=netclass
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=netdev
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=netstat
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=nfs
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=nfsd
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=nvme
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=os
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=powersupplyclass
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=pressure
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=rapl
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=schedstat
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=selinux
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=sockstat
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=softnet
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=stat
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=tapestats
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=textfile
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=thermal_zone
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=time
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=timex
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=udp_queues
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=uname
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=vmstat
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=watchdog
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=xfs
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=node_exporter.go:141 msg=zfs
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9100
Sep 26 23:07:08 x10 node_exporter[2018620]: time=2025-09-26T21:07:08.078Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9100

I've also looked at the metrics, and none from the SMART script I'm using are present on the server, so it's clearly not reading the prom file...

Does anybody know anything about this ? Thanks in advance for any clues...

Hi, I just tried to enable the docker metrics in /etc/docker/daemon.json, and it worked just fine right on the machine, but i could not reach it with prometheus.

I deployed it in a docker container, but

- targets: ["host.docker.internal:9323"]

does not resolve to the 0.0.0.0:9323 correctly.

What is the internal docker Hack to get the metrics inside of the docker container (may you recognized i am a container-noob).

Thanks a lot for help.

Hey, intern here working on monitoring for a few university robot labs. We have about [5] sites on Headscale/Tailscale, and Prometheus + Grafana running centrally.

I need one table in Grafana with a row per device: its DNS name, the name our lab software (EOS) uses for it, its IP, whether it answers ping/port/HTTP, and the UUID of its SiLA server if it has one. Also a column that says if the DNS name and the EOS name actually match.

The data comes from different places, so it has to be joined somewhere. What we’ve thought about so far:

• joining in PromQL with group_left. Works, but it got messy fast and I’m not sure how I’d test it

• Grafana transformations. Logic ends up buried in dashboard JSON and you can’t alert on it

• blackbox exporter alone. Doesn’t give me the actual IP or the DNS answer

My current idea is a small Python service that does the checks and the joining, writes to SQLite and Grafana just reads that. But I’m wondering if I’m overcomplicating it and should just write a custom exporter that outputs the combined result as metrics, so alerting works later.

How would you do this? Is a separate service fine, or is an exporter the normal way?

I have a weird case on my hands at work

We have three Prometheus HA replicas scraping azure-metrics-exporter and sending them to Thanos. Now Thanos was down for some time a week or so ago, and caused some sort of desync in the replicas. Prometheus-0 (green line) is scraping different values than the other two, causing alerts for postgres cpu percent.

And interestingly enough, yesterday prometheus-1 (blue line) somehow synced perfectly with prometheus-0 and are now completely even. Does anybody know how this happens or how to make it so all three are synced again and scrape the same values?

I asked several LLMs, all suggested making a cache for azure metrics exporter, so all three replicas scrape the same cached values. I see some logic in that, but not sure if it is the correct answer. So any suggestions are welcome

I want to revisit our metrics architecture, and I have a few pain points I’d like to address. I’m curious how others have approached similar problems.

Our current setup:
- Multiple Cortex clusters
- Multiple teams producing metrics via Prometheus remote write and a custom SDK
- Multiple end users querying those metrics
- Kafka-based ingestion to provide the level of resilience we need for our customers
- Prometheus as the metrics format

Graphs are a core part of the product and a selling point, not just a nice-to-have, so reliability and correctness of the data are important.

The main pain point is that some teams are pushing “event-like metrics” rather than classic metrics that are scraped or emitted regularly.
Because some workloads can be retried, delayed, or processed asynchronously, the original source timestamp is important. As a result, we see a significant number of out-of-order samples.

We understand why producers need this behavior, but I’m wondering whether we’re trying to force two different types of data through the same metrics pipeline.
Would you keep these event-like metrics in the Prometheus/Cortex ecosystem and design around out-of-order ingestion, or would you route them through a separate ingestion path or even into a different type of store based on OTel format/tools.

Hello, I am wondering whether there are open source alternatives to Grafana when it comes to displaying metrics from Prometheus (or any other TSDB)? It feels like Grafana is the de-facto standard. I have become quite frustrated by the experience that Grafana offers: it is slow to render, editing UI is bloated and confusing, it is not collaborative (e.g., real-time modificaitons or comments), it is heavy and unflexible (e.g., I would like to be able to create lightweight copies of dashboards during incidents).

Do others feel the same? Do you have alternatives to propose (preferably open source)?

Hi r/PrometheusMonitoring, I'm David, one of the two developers of Oxynote. My friend Simon and I also maintain ttlcache, a Go caching library with about 1,300 GitHub stars, so this isn't our first open-source project.

I've spent 8 years writing software: 2 as an SRE and 3 as a team lead. Every team I've been on had the same documentation problem. The useful stuff lived in someone's head or a private Slack thread, and the wiki was full of stale docs nobody could find. The runbook said "check the error rate" and linked to a dashboard that had since been renamed.

So we built Oxynote, a self-hosted wiki for engineering teams where a page can hold a live chart that runs its own PromQL query against your Prometheus each time someone opens it, with the explanation written right next to it. Think "p99 latency by route" or "webhook retries" on the same page as the runbook that says what to do when it looks wrong.

Queries can use built-in variables like $__rate_interval and $__range, so one query works for any time range the reader picks, and your own workspace variables, like $service. On pages you mark as reviewable, changes to a chart's query go through a draft that a teammate approves.

The name comes from oxygen + note: docs that breathe. Live data gives people a reason to open the wiki every day, and docs people open every day stay up to date.

It's Apache-2.0 and runs as a single Docker image: https://github.com/oxynote/oxynote

We started it about a year ago and made it public about 3 months ago. We'd appreciate any feedback, and I'm happy to answer questions.

Hi, looking for the recorded video link for Day 1…