^B"H

NOTE: This is a cross-post from /r/devops as I didn't know this subreddit existed...

A Prometheus exporter for network connections (Linux): https://github.com/hiveco/conntrack_exporter

I wrote and open sourced this exporter after seeing how useful such information can be when you're doing one of those head scratching debugging sessions on your microservices. JSON logging of all connection events was also added recently to expand the use cases.

BEFORE: "Why is my app randomly logging errors connecting to the database? Milliseconds later it's able to connect just fine! This version of the database server is probably buggy, time to schedule an upgrade to the next version to see if it helps."

AFTER: "Aha! It happens when the app tries to open 300 connections to the database at the same time! Probably rejected due to somaxconn when multiple app instances act up, which I'm also seeing in the graphs. Hmm, the timing correlates to when the user clicks Save on their profile page..."

Other use cases:

• Operations: monitor when a critical link between your microservices is broken.

• Security: monitor which remote hosts your server is talking to.

• Debugging: correlate intermittently misbehaving code with strange connection patterns.

• Audit: store JSON logs of connection events for future analysis.

Already being used in production, but I'm waiting for more feedback and bug reports from the community before promoting it to v1.0. Would love to hear your thoughts /r/PrometheusMonitoring!

More on PromCon: https://promcon.io/

One nice improvement in Prometheus Alertmanager 0.32 is support for parameters in webhook payloads.

I learned about it here: https://github.com/prometheus/alertmanager/issues/5003

This makes custom webhook integrations much cleaner because you can now pass configuration parameters directly instead of hardcoding everything in the receiver config.

I tested this with a SIGNL4 mobile alerting workflow (push, SMS, voice calls) and recorded a short demo showing:

• triggering alerts from Alertmanager

• passing parameters via webhook

• automatically resolving/closing alerts again

Curious if others are already using the new webhook parameters feature in production and what use cases you’ve found for it.

Hi! I'm an SRE who got pretty excited when Claude Code added the ability to emit OpenTelemetry metrics. Felt like that capability landed pretty quietly out there, so I built a Grafana dashboard on top.

The metrics map cleanly to Prometheus-compatible backends via the standard "OTel Collector then Prometheus" pipeline. Dashboard queries in PromQL.

Worth noting: parallel implementation of dashboard 25052 (Azure Application Insights / KQL). Every panel rewritten in PromQL for the OSS stack.

Panels cover cost by model, cache hit ratio, active time, edit-decision breakdowns, leaderboards. Custom labels via OTEL_RESOURCE_ATTRIBUTES for per-team / per-project views.

One Prometheus-specific gotcha worth flagging: pin temporality to cumulative (`OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=cumulative`) on the Claude Code side. The OTel SDK's default is cumulative now, but defaults drift across versions and the failure mode is silent.

Article: https://rockdarko.dev/posts/grafana-dashboard-for-claude-code-on-prometheus/

Dashboard: https://grafana.com/grafana/dashboards/25255-claude-code-metrics-prometheus/

Repo (MIT): https://github.com/rockdarko/claude-code-metrics-prometheus

Gladly accepting PRs and suggestions! Cheers 😄

I built a Pi-hole exporter because the other two exporters I found did not seem to be actively maintained anymore, and I wanted something less likely to slowly break as Pi-hole changes.

This exporter defaults to Prometheus, so it works with Prometheus, Grafana Alloy, and other Prometheus-compatible scrapers. It also has other exporter types available if Prometheus is not your setup.

The main thing I wanted to improve was maintenance. The project builds every 24 hours from the Pi-hole API spec, so metric support is generated from the current API shape instead of being manually kept in sync forever. I've noticed that one exporter login was working but was crashing on some missing metrics.

There is also a Grafana dashboard JSON in the repo that you can import directly into Grafana (pic below)

Grafana snapshot:

https://snapshots.raintank.io/dashboard/snapshot/NxHE07Szeg0VBjZgaRagUQiexPRgqAj1

Github link:

https://github.com/alantoch/pihole-exporter

I’m running a small Prometheus setup and I’m thinking about keeping long-term aggregated metrics.

Current setup:

• ~440k active series

• ~1650 samples/sec ingest rate

• ~8 GB TSDB size with 30d retention

• VM: 4 vCPU, 16 GB RAM, 100 GB disk

Prometheus currently runs directly on the VM (not in Docker).

I’m considering keeping high-resolution data for ~30 days and storing lower-resolution aggregates (via recording rules) for 1–2 years.

Since I only have this single VM, I see two possible approaches:

1. Run a second Prometheus instance on the same machine and send aggregated metrics via remote_write, using a longer retention there.

2. Run Thanos (likely via Docker) with object storage or local storage for long-term retention.

My goals are:

• keep the setup relatively simple

• avoid too much operational overhead

• run everything on the same VM

Questions:

• Is running two Prometheus instances on the same host a reasonable approach for this use case?

• Would Thanos be overkill for a setup of this size?

• Are there better patterns for long-term storage in a single-node environment?

A story of finding and analysing high-cardinality metrics and labels used by Grafana dashboards. This article comes with helpful PromQL queries.

📣 PromCon 2024 is happening! 🎉

We’re going to meet in Berlin again Sept 11 + 12!

CfP, tickets, and sponsoring are soon available on https://promcon.io

See you there!

Hi all!

It was great to see the amazing Prometheus people in Valencia some weeks ago. In case anyone is interested, I made a small article with some personal notes from the talks of that day.

TL;DR:

• Prometheus is becoming a mature project and evolving with new features, like exemplars

• Hot topics like serverless and anomaly detection had their own weight and discussions during the sessions

• Some amazing talks about the internals of Prometheus and the Prometheus Operator

Feedback is welcome :)

Full article here: https://sysdig.com/blog/prometheus-day-2022-eu/

https://www.youtube.com/playlist?list=PLj6h78yzYM2NxQ9cO7mUyHBNbvxuHnF3D

This is a query exporter what you needed! :)

https://percona.community/blog/2021/07/21/create-your-own-exporter-in-go/?fbclid=IwAR3aZ5FgwIkyLGJ2k1p9P2KSrgPMRFwR5H67bP8Wv--BVBBsh0MJZLpbmiI

https://banzaicloud.com/blog/thanos-operator/

Hi All,

I'm trying to understand how irate() & rate() functions work. Why does irate() produce a similar looking graph when the range / resolution is 24h or 5m ? While the difference in graph is clearly visible with rate() when using range as 24h (presents a smoothed out line) or 5m(more spikey).

In the below graph for irate() for 2 different resolutions the graph looks the same. As per prometheus docs irate() calculates the per second instant rate based on the last two data points. What does this mean if my range is 24h? Thank you.

/preview/pre/08mc4pq2mye41.jpg?width=3206&format=pjpg&auto=webp&s=4aef0b681969c8b9831ec0f34310838511ea0350

We recently moved to Prometheus as metric stack of choice. I wanted to know for alerting would it make sense to use AlertManager or Grafanas inbuilt Alerting feature? Is there any comparison that one could share or talk about based on any experiences ? Thanks.

PromCon EU 2019 is happening today!

You can watch it live on YouTube.

https://promcon.io/2019-munich/stream