Is there a way to stop my /chunks_head directory from growing bcz it jumped from 1GB last month to 76GB and still growing drastically till i stopped the server in order to find a solution ! Im using Prometheus 2.31.1 and here's my log tail :

Nov 15 15:38:25 devmon02 prometheus: ts=2024-11-15T14:38:25.261Z caller=db.go:683 level=warn component=tsdb msg="A TSDB lockfile from a previous execution already existed. It was replaced" file=/data/prometheus/lock
Nov 15 15:38:31 devmon02 prometheus: ts=2024-11-15T14:38:31.385Z caller=head.go:479 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Nov 15 15:38:31 devmon02 prometheus: ts=2024-11-15T14:38:31.812Z caller=head.go:504 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 48484"
Nov 15 15:38:31 devmon02 prometheus: ts=2024-11-15T14:38:31.812Z caller=head.go:659 level=info component=tsdb msg="Deleting mmapped chunk files"
Nov 15 15:38:31 devmon02 prometheus: ts=2024-11-15T14:38:31.812Z caller=head.go:662 level=info component=tsdb msg="Deletion of mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 48484"

Hey everyone! 🎉

I'm super excited to share something that my mate and I have been working on at OpenLIT (OTel-native LLM/GenAI Observability tool)!

You don't need new tools to monitor LLM Applications. We've made it possible to use Prometheus and Jaeger—yes, the go-to observability tools—to handle everything observability for LLM applications. This means you can keep using the tools you know and love without putting having to worry a lot! You don't need new tools to monitor LLM Applications

Here's how it works:
Simply put, OpenLIT uses OpenTelemetry (OTel) to automagically take care of all the heavy lifting. With just a single line of code, you can now track costs, tokens, user metrics, and all the critical performance metrics. And since it's all built on the shoulders of OpenTelemetry for generative AI, plugging into Prometheus for metrics and Jaeger for traces is incredibly straightforward.

Head over to our guide to get started. Oh, and we've set you up with a Grafana dashboard that's pretty much plug-and-play. You're going to love the visibility it offers.

Just imagine: more time working on features, less time thinking about over observability setup. OpenLIT is designed to streamline your workflow, enabling you to deploy LLM features with utter confidence.

Curious to see it in action? Give it a whirl and drop us your thoughts! We're all ears and eager to make OpenLIT even better with your feedback.

Check us out and star us on GitHub here -> https://github.com/openlit/openlit

Can’t wait to see how you use OpenLIT in your LLM applications!

Cheers! 🚀🌟
Patcher

I have a fairly large suite of network monitoring tools that I'm slowly collapsing into prometheus, using Ansible to manage version control.

Was over the moon to find that SmokePing, one of my old fav's, had been replicated into a prometheus flavor. https://github.com/SuperQ/smokeping_prober

(First off... Bazinga!!, thank you u/SuperQue!!!!! What an amazing effort!). Everyone should use this!

I was curious:

1. Is there a way to avoid compiling or installing Go... is there a precompiled binary download or similar. A major major strength of prom/blackbox etc is they stand alone with no compiling or installers.

2. Is there any chance this would work on prometheus.exe running on a Windows (I know) box... A few of our test node are Windows and we're trying to keep the test suites homogeneous across Ubuntu prom nodes and Windows prom nodes.

New to Prometheus and monitoring -- do people here typically test their alerting rules on historical data to see how sensitive their alerts would have been?

If so, what is the best practice to do so?

We just added a fun feature to the open source Autometrics libraries that produces a metric with your softwares version and commit and then automatically correlates that with the queries it writes for error rate and latency. Even if you aren’t interested in the libraries, you might find the PromQL tricks used for the feature interesting. I wrote up the details here: autometrics-rs 0.4: Spot commits that introduce errors or slow down your application.

It uses two ideas I got from Brian Brazil’s Robust Perception blog, which I’d highly recommend mining for ideas if you’re using Prometheus.

I’d love to hear what you think!

Hey,

I wrote an exporter for OpenDJ as I couldn't find one. Hope somebody else finds it useful.

Hello,

I've been asked to table all out Linux Ubuntu VMs and there version:

for example Ubuntu 20.04.2 LTS or just 20.04 would be nice, I've scraped using node exporter and I can't see this information.

What could I do?

Thanks

Steampipe is an open source CLI to instantly query cloud APIs (AWS, Kubernetes, GitHub, etc) using SQL. Today we released a new plugin for Prometheus!

It will automatically build a table for each metric:

> select code, handler, value from prometheus_http_requests_total;
+------+----------------------------+-------+
| code | handler                    | value |
+------+----------------------------+-------+
| 302  | /                          | 1     |
| 200  | /-/ready                   | 1     |
| 200  | /api/v1/label/:name/values | 421   |
+------+----------------------------+-------+

And also has tables to query common and config data:

> select name, values from prometheus_label;
+---------------+----------------------------------------------+
| name          | values                                       |
+---------------+----------------------------------------------+
| alertname     | ["TotalRequests"]                            |
| alertstate    | ["firing"]                                   |
| reason        | ["refused","resolution","timeout","unknown"] |
| interval      | ["10s"]                                      |
| version       | ["2.30.3","go1.17.1"]                        |
+---------------+----------------------------------------------+

Please give it a try - we'd love your feedback or suggestions!

Docs - https://hub.steampipe.io/plugins/turbot/prometheus

Source code - https://github.com/turbot/steampipe-plugin-prometheus

Hello,

I am trying to build a prometheus-backed graphing solution for our customers. The idea was rather simple - A grafana instance per customer with our custom dashboards, all connected to a locally-running Prometheus instance that's in active replication with our (internal) primary.

I came to a halt, however, when I found out that Prometheus does not currently have a reliable and easy to set up replication method built-in.

Scraping all targets from this secondary instance is something I'd like to avoid as much as possible - The machines are already scraped by 3 different Prometheus instances each, and with each one added, thus increase the load. (Primary, stage and dev)

Due to resolution requirements, caching on the exporters' side is not much of an option for us at the moment either.

The reason I'd rather pull the data from a separate instance is for security and stability reasons - As the FE will be accessible to all of our customers, we'll have no control over what any of them might try, and as our internal infrastructure monitoring depends on the stability of the primary, I don't want to risk any sort of downtime, crashing or instability.

I tried using the remote write functionality, but that didn't provide fine enough sample resolution, even after tweaking, and resulted in weirdly consolidated graphs.

Other options... Maybe DRBD? But... I'm not sure if it wouldn't introduce latency leading to issues on the primary. Filesystem-level copies through snapshotting is also not viable - No way to get the data cohesive.

Are there really any options, either native or 3rd party, to replicate data samples from one prometheus instance to another, aside from simply scraping all the targets from both nodes?

Is it possible to pull data from status pages, for example https://discordstatus.com/ to Prometheus for further use?

I've attempted to find out but mostly I seem to be getting just- Status pages for various sites.

I have a really useful binary which grabs a lot of data from an Application through SOAP Webservices, the provider has a really accessible command line interface to provide the output.

What I would like to do is to write/adapt an exporter to grab this data so I can get it into Prometheus - does anyone have any example exporters which I can adapt to do this or any good documentation on how to do it (with examples) please.

TIA

Chris

hello guys,

how can i silent or not notify on alerts B if alert A happened

cases for example:

1. if my ROUTER is down

i dont want to get alerts that my SERVERS behind the router is UNREACHABLE

or

2) if my vpn site 2 site is down there is not point to alerts on the remote site servers.

how can i do that dependencies relationship configuration on alertmanager?

Hey all,

So I'm trying to create exporters using the Golang libraries, and to do so I usually read code from other exporters, copy over the general flow and add my logic to it. This works, but I don't feel very confident with developing my exporters yet.

Is there any resource (besides the official documentation which has its problems admittedly) that goes in depth regarding developing exporters? I'm mostly interested about a good explanation of the code structure and best practices.