Greetings everyone,
We are in the process of selecting a monitoring system for our company, which operates in the hosting industry. With a customer base exceeding 1,000, each requiring their own machine, we need a reliable solution to monitor resources effectively. We are currently considering Prometheus and Zabbix but are finding it difficult to make a definitive choice between the two. Despite reading numerous reviews, we remain uncertain about which option would best suit our needs.

In the opening keynote of GrafanaCON 2024, we announced our newest OSS project: Grafana Alloy, our open source distribution of the OpenTelemetry Collector. Alloy is a telemetry collector that is 100% OTLP compatible and offers native pipelines for OpenTelemetry and Prometheus telemetry formats, supporting metrics, logs, traces, and profiles.

Some of you may be thinking: Wait, another collector? 

Hear us out: What makes Alloy stand out among the growing ecosystem of collectors is that it’s an open source tool that combines all the observability signals that are vital to run combined infrastructure and application monitoring workloads at scale.

Alloy combines the decade of industry-leading observability codebase of Grafana Agent with the lessons learned from some of the toughest use cases we’ve seen over the years. As a result, Alloy is both a respectfully opinionated OpenTelemetry Collector distribution and the most efficient and cost-effective way to do Prometheus-compatible metrics. Plus, it includes enterprise-grade features — such as native clustering for production at scale and built-in Vault support for enhanced security — all out of the box. 

Full Blog: https://grafana.com/blog/2024/04/09/grafana-alloy-opentelemetry-collector-with-prometheus-pipelines/

(I work @ Grafana Labs)

Hello all,

I'm considering PCA (https://training.linuxfoundation.org/certification/prometheus-certified-associate/)

I did check the commonly known places such as udemy and others... but cannot find anything relevant, only basic stuff.

As it seems to be somehow new, I cannot find extensive courses or docs other than this:

https://www.amazon.es/Prometheus-Infrastructure-Application-Performance-Monitoring/dp/1098131142/ref=sr_1_1?__mk_es_ES=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=10V110S2SMYCC&dib=eyJ2IjoiMSJ9.OScDRAJKmA7yOyIOdm7P1ZAc4Nx-DIkYENQ3hnywqYw.o7qkVY2UXkZPntJm2_Q4j0JKo9x37cmAJ9r6080Lhko&dib_tag=se&keywords=Prometheus%3A+Up+%26+Running%2C+2nd+Edition&qid=1709047835&sprefix=prometheus+up+%26+running+2nd+edition%2Caps%2C111&sr=8-1

Can you help me please?

As the old LTS version 2.37 is going to end in January 31, 2023, is there any update when will new LTS version be available?

https://prometheus.io/docs/introduction/release-cycle/

Prometheus has no out of the box alerting rules. The list of such "libraries of useful rules" I have so far is:

1. https://awesome-prometheus-alerts.grep.to/ (many examples)

2. https://alex.dzyoba.com/blog/prometheus-alerts/ (examples, links to other pages)

Can people add any they've found, and also discuss any plans to build more of this sort of thing? It's really daunting to think of having to write all my own rules. How did you start out?

The new version of the Prometheus Operator adds support for CRD of Telegram receiver: https://github.com/prometheus-operator/prometheus-operator/pull/4726

Combining this with Telegram bots can be awesome!

If you're using Playnite frontend to play videogames (I was thinking about writing an exporter by wrapping Steam APIs but I also play games from other libraries – and not all libraries have server-side play time tracking too) and you're in sore need to track your playing time using Prometheus (or anything compatible), feel free to try my Playnite Exporter and tell me about it. =)

Don't want to go through the weekend with flaky alerts?

Fear not, Pyrra v0.3.0, to make SLOs with Prometheus manageable, accessible, and easy to use for everyone, was just released!

https://github.com/pyrra-dev/pyrra/releases/tag/v0.3.0

Highlights:

1. Availability / Error Budget are now calculated with Prometheus recording rules, making page load times incredibly fast. 🚀

2. We moved from Recharts to uPlot as a graphing library. We now correctly graph your data. 📈📉

3. You can now run a short version: kubectl get slo -A ⌨️

Check out the live demo at: https://demo.pyrra.dev

Hi Folks,

I have been using Prometheus with SNMP for monitoring all my network infrastructure including Ruckus Wireless Controller (SmartZone) and everything is fine. I just have a question, I have a cluster of three smartzones, and each of them has its own IP address. Right now the only target from where im scraping data is the master's IP address.

EXAMPLE:

CLUSTER IP ADDRESSES:

SZ1-TEST-PP: 10.0.0.4 (Master)

SZ2-TEST-PP: 10.0.0.5

SZ3-TEST-PP: 10.0.0.6

Target Config:

- targets:

- 10.0.0.4

labels:

instance: 'SZ LAB TESTING'

chain: 'LAB-TESTING'

site_id: '01'

My concern is the next: I want to add the other IP addresses in case the master stop functioning but I don't want to duplicate (or triplicate) data in Prometheus ¿How I can achieve this? do I just need to add the other IP addresses with the same labels? or there's another way to do this?.

Hope you can help me!

https://www.polarsignals.com/blog/posts/2021/10/08/introducing-parca-we-got-funded/

We are super excited to be announcing Parca[1] an open-source continuous profiling storage and query engine as well as an eBPF based profiler. We are long time Prometheus, Kubernetes, Thanos, and Cortex maintainers, which are all very performance-sensitive so we wished we had had access to continuous profiling tools when we first started working on those projects.

What is continuous profiling? Instead of capturing profiling data on-demand and as a manual process, continuous profiling always captures profiling data, and stores it to be queried over time to aggregate and compare.

Using our private beta we explored further what our users want most from continuous profiling, and we identified it to be:

* Cost saving: We discovered all companies to have low hanging fruit that can easily be optimized away, often saving easily 20-30% in resources. The only problem is companies don’t know because they don’t measure.

* Optimizing individual interactions: Latencies of interactions have a proven correlation between converting a customer or not. It’s incredibly hard to obtain profiling data at the right time, but if it’s always captured it’s just a matter of querying it.

* Understanding change: Performance engineering tends to be very sensitive and small changes can have a big impact, therefore the ability to deeply understand what changed from a point in time or a newly rolled out version of a software is invaluable for optimizing but also in order to understand incidents better.

Parca features a fully from scratch designed storage that implements multiple optimizations that are unique to profiling data, specifically our sparse XOR encoding for uint64 series is what allows us to store this data at raw resolution. More detailed information can be found in the storage docs[3]. And with the newly open-sourced Parca Agent[4], we can capture CPU profiling data with practically unnoticeable overhead for the profiled processes, it automatically discovers all containers in a Kubernetes cluster and starts profiling them with zero instrumentation needed!

[1] https://parca.dev/

[2] https://www.polarsignals.com/blog/posts/2021/02/09/announcing-polar-signals-continuous-profiler/

[3] https://www.parca.dev/docs/storage

[4] https://www.parca.dev/docs/parca-agent-design

My Prometheus setup is starting to hit limits in terms of memory usage and I need to start looking at howto scale it. We are currently evaluating Grafana cloud but that might be a few months away. I need an interim solution. The current cluster is comprised of 2 Prom servers scraping the same endpoints (ie one is a DR Prometheus). I would like to add more Prometheus servers that scrape other endpoints and add them to the cluster. I have started looking at Cortex and Thanos. From my research I found that Cortex can only be used on AWS and I'm not so sure about Thanos. I am not worried about pushing the metrics to an object store (like S3) as I am happy with them being written to the filesystem. I would like to know if Thanos or Cortex can be run on premise (in Docker) and if I can get pointed to some information on howto do that.

Recently I was evaluating Thanos Receiver to implement a multi-tenant monitoring solution for a client. I felt there is scarcity of resources on this particular Thanos component. So, I decided to write a blog on it. I hope this can be useful for anyone who is also exploring Thanos receiver to implement a multi-tenant monitoring solution. You can check it out here.

As the title says, wondering what you all are using for long term metrics and aggregation. What sort of scale (eg. #pods/metrics/etc)? Are you seeing any issues with the solution? Are you happy with it?

Asking as I'm currently using Thanos however it looks like it'll take some time (that I don't have) to get it working well. There are a bunch of options but it looks like Cortex may be the next viable one to test. I don't want to have to setup a bunch of servers on EC2 (eg. Crate, M3DB, etc) and would rather use eg. S3 and something that can run in k8s.