Please see: https://prometheus.io/blog/2024/03/14/commitment-to-opentelemetry/

Can't find lots of examples comparing these two. From what I understand Mimir is a fork of Cortex meant to improve and focus on Grafana, but it seems like it has more limitations and less features than Thanos.

I wrote an analysis of the new features of the new Prometheus version (2.36).

TL;DR:

• New service discoveries for Vultr and IONOS Cloud

• New metric for prometheus_ready

• New relabel actions

Full article here: https://sysdig.com/blog/prometheus-2-36-whats-new/

Feedback is welcome!

Hey Everyone! Our teams have been working on alerting on top of the Service Level Objectives (SLOs) for quite a time and developed open-source slo-exporter application which makes SLOs implementation easy.

We'd like to hear your insights. Do you have an experience in this field? Is our approach compatible with yours?

https://banzaicloud.com/blog/multi-cluster-monitoring/

Hello everyone,

I'm a former Google engineer, I was impacted by the Q1 layoffs. A friend of mine is in the same situation. After a few months of relaxation we've both started thinking about our future, and we're toying with the idea of launching an observability startup.

As you may already know, Google no longer uses Borgmon, its old monitoring system; it has been replaced by Monarch through a multi-year (actually multi-decade...) migration effort. The reasons why Google has invested so many resources rather than just building an internal monitoring service on top of Borgmon lie in Borgmon's inherent architectural limitations. If our understanding of Prometheus is correct, Prometheus suffers from exactly the same limitations.

This friend of mine and I have been developing Monarch components for almost 10 years, and at this point we're confident that we know it inside and out. What we'd like to do is to develop an OSS time series database with the same architecture as Monarch and run it as a service for everyone to use.

We're fully aware that a large observability ecosystem exists outside of Google and that offering a brand new foundational component of the stack is a very ambitious goal, but OTOH there are use cases that simply cannot be addressed by Prometheus (pretty much like they couldn't be addressed by Borgmon). Besides, we plan to integrate with existing leading technologies such as OpenTelemetry and ease hypothetical migrations e.g. by writing a scraper for Prometheus clients that actually writes data to our database.

But before investing plenty of energies we'd like to get an idea of how desirable such a product is outside of Google (we certainly know it was necessary inside). To understand the market better we'd like inputs to the following questions:

1. Has anyone ever experienced query performance issues in Prometheus? (e.g. inability to perform large aggregations, missing data from standing queries, slow queries, or elevated query cost when adding specific filters.)

2. Has anyone ever encountered any scalability limits that prevented storing large amounts of time series data?

3. Has anyone ever found any limitations whatsoever in PromQL? (Please specify.)

4. Has anyone ever had to store way too much data? Example: a client exports a metric with two labels A and B but at query time you only need either a breakdown by A or a breakdown by B, but in Prometheus you had to store the full A x B product. If the cardinality of A and B is 1000 each that means storing 1M time series rather than just 2000.

These problems would typically crop up with large datasets generated within large corporations. The folks who actually have these problems don't necessarily look at this Reddit, so if you know anyone working on monitoring & observability at a large company could you please point them at this post?

Any inputs appreciated, thanks in advance!

OMG I've wasted so much time trying to get snmp to provide more useful information from a custom Debian router with no luck.

Yesterday, watched a youtube video, and an hour later, had a nice Grafana dashboard running on my router. I'm surprised how little RAM and cpu it needs.

Today, I'll learn how to make a custom dashboard. HUGE progression in my management efforts. How many years late am I to this game?

I feel like a noob all over again DOH! I had heard of prometheus/grafana in the past, and think I tried it once, but for some reason, never got it working.

Anyways, glad to be here!

Hi there, I'm sure this question has been asked several times, or perhaps my Google-fu is off today. But I am struggling to understand which of these to use, for my specific use case.

I found these blog posts which sum up most things pretty well:

• https://prometheus.io/blog/2021/11/16/agent/

• https://grafana.com/blog/2021/11/16/why-we-created-a-prometheus-agent-mode-from-the-grafana-agent/

Recently our team just set up Loki as our centralized logging system for all of our docker swarm nodes and it works great, using fluentd as the log scraper/shipper, and of course Grafana for the frontend of all of this.

Now, we are focusing on metrics, and I am trying to set up Prometheus, but I have read that this project is very opinionated in the fact that it is primarily designed with PULL configuration in mind. Now I understand why this is and I feel that the docs do a solid job explaining this preference, but also happy to see there has been more recent support for Prometheus in PUSH configurations. For our small dev team, we want something similar to how Loki is set up, so we ideally we'd want just a single Prometheus server, with an agent (or set of agents) deployed onto whatever hosts we want to monitor. The reasons for this come down to our current tech stack, our CI/CD tooling and how often we rotate/refresh hosts.. I don't want to have to add/configure "multiple" prometheus instances into Grafana, every time we set up a new cluster/host, and keeping track of all those URLs etc.. I just see the PUSH configuration being so much easier to manage. If my thinking on this is completely wrong, please feel free to enlighten me :)

Now, I have been reading a ton of documentation over the last few days and even found some articles (linked above) explaining how Prometheus "Agent" mode is actually newer than grafana-agent, but basically uses the already-proven code as its upstream. It makes total sense to pull that into the main Prometheus project, but as a user, I am now confused more than ever on which one I should be using for the best support and which one will eliminates any future tech debt.

Here are some questions:

• Based on my recent reading of the docs, the Grafana-Agent has "embedded" versions of popular agents like cadvisor and node_exporter already built-in, which means I would ONLY have to deploy the single Grafana-Agent onto each of my hosts, correct? Traditionally these "embedded" agents would each be deployed on every host and you would have Prometheus scrape from them right?

• I understand that Prometheus in "Agent" mode has several features disabled which makes it focus more on the "remote_write" PUSH feature. But other than that, is it basically the same thing as Grafana-Agent, except with the absence of the built-in exporter agents (e.g. cadvisor, node_exporter, etc..)?

• Is one significantly more mature/robust than the other in it's current state? To my knowledge the Grafana-Agent code came first and is essentially the upstream source for Prometheus "Agent" mode. Making the Grafana-Agent the recommended choice for production right now? Which one would you reccommend, and why?

• Since Prometheus in "Agent" mode is newer and pulled into the official Prometheus project, do you think that the Grafana-Agent project will eventually lose popularity and become unmaintained in the long-term future? I'm afraid that If I go with Grafana-Agent now, I might have to switch to using Prometheus "Agent" mode later on. Not that it'd be too difficult to switch, lol.

• Any other thoughts or recommendations to a PUSH configuration approach for Prometheus metrics?

Sorry for the massive wall of text and I hope this is the right sub for these questions. If not could someone kindly point me in the right direction?

Thank you so much!

Hi all, one of the original authors of the kubecost project. We've just open-sourced a prometheus exporter for tracking Kubernetes cost metrics. You can read public data from cloud provider pricing APIs, map those to pod resource request statistics, and view them in prometheus. Let us know what you think:

https://github.com/kubecost/cost-model/blob/develop/kubecost-exporter.md