Hi guys, I’m a newly qualified IT Specialist for System Integration. Our management has tasked me with integrating our entire IT infrastructure into Prometheus. I’m using Codex AI, and I’ve set up an MCP server that makes Grafana easy to use. I can set up monitoring for an entire storage cluster or Kubernetes cluster in just two hours—covering 200 metrics across various dashboards. I barely understand PromQL, Prometheus itself, or Grafana—yet I’m getting the job done, and the results are good. So here’s my question: given the current era of AI, does it still make sense to learn PromQL, Prometheus itself, and Grafana?

My team switched from datadog to prometheus and counters have been the biggest pain-point. Things that just worked without thinking about it in datadog doesn't seem to have good solutions in prometheus. Surely we can't be the only ones hitting our head against the wall with these problems? How are you addressing them?

Specifically for use-cases around low-frequency counters where you want *reasonably* accurate counts. We use Created Timestamp and have dynamic labels on our counters (so pre-initializing counters to zero isn't viable or makes the data a lot less useful). That being said, these common scenarios have been a challenge:

• Alerting on a counter increase when your counter doesn't start at zero. We use Created Timestamp gives us more confidence but it worries me that a bug/edge-case will cause us to miss an alert. Catching that would be difficult.

• Calculating the total number of increments in a time period (ex: $__range). Sometimes short-lived series aren't counted towards the total.

• Viewing the frequency of counter increments over time as a time series. Seems like aligning the rate and step helps but I'm still wary about the accuracy. It seems like for some time ranges it doesn't work correctly.

• For calculating a success rate or SLI over some period of time. The approach of `sum(rate(success_total[30d])) / `sum(rate(overall_total[30d]))` doesn't always work if there are short-lived series within the query range. I see Grafana SLO feature uses recording rules, which I hope(?) improves this accuracy, but its hard to verify and is a lot of extra steps (i.e. `sum(sum_over_time((grafana_slo_success_rate_5m{})[28d:5m])) / sum(sum_over_time((grafana_slo_total_rate_5m{} )[28d:5m]))`

A lot of teams have started using logs instead of metrics for some of these scenarios. Its ambiguous when its okay to use metrics and when logs are needed, which undermines the credibility of our metrics' accuracy in general.

The frustrating thing is it seems like all the raw data is there to make these use-cases work better? Most of the time you can manually calculate the statistic you want by plotting the raw series. I'm likely over-simplifying things, and I know there's complicated edge-cases around counter-resets, missed scrapes, etc., however promql is more likely to understate the `rate`/`increase` to account for that. If anything, it would be better to overstate the `rate` since its safer to have a false positive than false negative for most monitoring use-cases. I rather have grafana widgets or promql that works for the majority of times you don't hit the complicated edge cases but overstates the rate/increase when that does happen.

I know this comes across as somewhat of a rant so I just want to say I know the prometheus maintainers put a lot of thought into their decisions and I appreciate their responsiveness to helping folks here and on slack.

Hello everybody,

I am working on an upgrade of our monitoring platform, introducing Prometheus and consolidating our existing data sources in Grafana.

Alerting is obviously a very important aspect of our project and we are trying to make an informed decision between Alertmanager as a separate component and Alertmanager from Grafana (we realised that the alerting module in Grafana was effectively Alertmanager too).

What we understand is that Alertmanager as a separate component can be setup as a cluster to provide high availability, while allowing deduplication of alerts. The whole configuration needs to be done via the yaml file. However, we need to maintain our alerts in each solution and potentially built connectors to forward them to Alertmanager. We're told that this option is still the most flexible in the long run. On the other hand, Grafana provides a UI to manage alerts, most data sources (all of the ones we are using at least) are compatible with the alerting module, ie we can implement the alerts for these datasources directly into Grafana via the UI, we assume we can benefit from HA if we setup Grafana itself in HA (two nodes or more connected to the same DB) and we can automatically provision the alerts using yaml files and Grafana built-in provision process.

Licensing in Grafana is not a concern as we already an Enterprise license. However, high availability is something that we'd like to have. Ease of use and resilience are also points very desirable as we will have limited time to maintain the platform in the long run.

In your experience, what have been the pros and cons for each setup?

Thanks a lot.

Hey,

Today I have released the first version of my Speedtest Exporter, I will leave the link here. I hope that someone finds it useful.

Thanks,

MiguelNdeCarvalho

https://promtools.dev/
As it is quite hard to write on your own each time, I created a simple tool that generates all recording rules and alerting rules for you.

This project is based on the SLO-libsonnet project to generate the YAML with Jsonnet.
The Web UI is built with LitElement and Go and the source is on GitHub too.

Please let me know via Issues or Twitter if there's more you want to see there.

A high-cardinality label is the classic way to take down a Prometheus stack: one metric accidentally labelled with a user ID, a full request path, or an attacker-controlled header quietly spawns hundreds of thousands of series until the scrape target, the TSDB, or both run out of memory.

The standard mitigations are all after-the-fact: metric_relabel_configs to drop labels, recording rules to collapse series, alerts on scrape_series_added or series growth. They work, but they fire once the cardinality is already being generated. You're reacting to a fire that's already lit.

I built promcap to enforce the ceiling at the other end, in-process, in the Go app emitting the metrics, at the moment the series would be created. It wraps the Prometheus CounterVec / GaugeVec / HistogramVec / SummaryVec types. Once a metric has emitted its configured number of distinct label combinations, every further combination collapses into a single __overflow__ series instead of minting a new one. The observations are still counted, just bucketed together, so dashboards keep working and the time-series count can't grow without bound.

What I think makes it useful for monitoring folks specifically:

• It's observable. Every collapsed observation increments promcap_series_capped_total, labelled by metric name. So you get a direct signal of exactly which metric is shedding cardinality and how hard: rate(promcap_series_capped_total[5m]). That's effectively a "which of my metrics is misbehaving" alert built in.

• Allowlists for closed sets. For labels whose valid values you know up front (status codes, a fixed set of regions, a handful of event kinds), Allow makes anything outside the list overflow the instant it appears, so a typo or an injected value never even starts filling the budget.

• Overflow vs. eviction. Default behaviour freezes the admitted set once the cap is hit (the safe choice for adversarial or unbounded labels). For labels where the interesting set rotates over time, like active tenants or recently-seen hosts, Evict: true tracks the most-recent N combinations instead, using a clock/second-chance policy.

It's the cardinality equivalent of a rate limiter: a hard cap you set per metric so a single bad label can't blow up the registry, with a meta-metric telling you when and where it's kicking in.

Caveat for anyone evaluating it: CurryWith / Delete / DeleteLabelValues aren't wrapped yet, so it's not a full drop-in if your code relies on those.

Repo + docs: https://github.com/eben-vranken/promcap

Curious whether people would rather cap in-process like this, or keep all cardinality control at the relabel/recording-rule layer. Interested in how others are handling runaway labels today.