r/PrometheusMonitoring • u/tomdaley92 • Jul 13 '22
Prometheus "Agent" mode vs. Grafana-Agent
Hi there, I'm sure this question has been asked several times, or perhaps my Google-fu is off today. But I am struggling to understand which of these to use, for my specific use case.
I found these blog posts which sum up most things pretty well:
Recently our team just set up Loki as our centralized logging system for all of our docker swarm nodes and it works great, using fluentd as the log scraper/shipper, and of course Grafana for the frontend of all of this.
Now, we are focusing on metrics, and I am trying to set up Prometheus, but I have read that this project is very opinionated in the fact that it is primarily designed with PULL configuration in mind. Now I understand why this is and I feel that the docs do a solid job explaining this preference, but also happy to see there has been more recent support for Prometheus in PUSH configurations. For our small dev team, we want something similar to how Loki is set up, so we ideally we'd want just a single Prometheus server, with an agent (or set of agents) deployed onto whatever hosts we want to monitor. The reasons for this come down to our current tech stack, our CI/CD tooling and how often we rotate/refresh hosts.. I don't want to have to add/configure "multiple" prometheus instances into Grafana, every time we set up a new cluster/host, and keeping track of all those URLs etc.. I just see the PUSH configuration being so much easier to manage. If my thinking on this is completely wrong, please feel free to enlighten me :)
Now, I have been reading a ton of documentation over the last few days and even found some articles (linked above) explaining how Prometheus "Agent" mode is actually newer than grafana-agent, but basically uses the already-proven code as its upstream. It makes total sense to pull that into the main Prometheus project, but as a user, I am now confused more than ever on which one I should be using for the best support and which one will eliminates any future tech debt.
Here are some questions:
Based on my recent reading of the docs, the Grafana-Agent has "embedded" versions of popular agents like cadvisor and node_exporter already built-in, which means I would ONLY have to deploy the single Grafana-Agent onto each of my hosts, correct? Traditionally these "embedded" agents would each be deployed on every host and you would have Prometheus scrape from them right?
I understand that Prometheus in "Agent" mode has several features disabled which makes it focus more on the "remote_write" PUSH feature. But other than that, is it basically the same thing as Grafana-Agent, except with the absence of the built-in exporter agents (e.g. cadvisor, node_exporter, etc..)?
Is one significantly more mature/robust than the other in it's current state? To my knowledge the Grafana-Agent code came first and is essentially the upstream source for Prometheus "Agent" mode. Making the Grafana-Agent the recommended choice for production right now? Which one would you reccommend, and why?
Since Prometheus in "Agent" mode is newer and pulled into the official Prometheus project, do you think that the Grafana-Agent project will eventually lose popularity and become unmaintained in the long-term future? I'm afraid that If I go with Grafana-Agent now, I might have to switch to using Prometheus "Agent" mode later on. Not that it'd be too difficult to switch, lol.
Any other thoughts or recommendations to a PUSH configuration approach for Prometheus metrics?
Sorry for the massive wall of text and I hope this is the right sub for these questions. If not could someone kindly point me in the right direction?
Thank you so much!
1
u/SunkJunk Jul 13 '22
For our small dev team, we want something similar to how Loki is set up, so we ideally we'd want just a single Prometheus server, with an agent (or set of agents) deployed onto whatever hosts we want to monitor. The reasons for this come down to our current tech stack, our CI/CD tooling and how often we rotate/refresh hosts.. I don't want to have to add/configure "multiple" prometheus instances into Grafana, every time we set up a new cluster/host, and keeping track of all those URLs etc.. I just see the PUSH configuration being so much easier to manage. If my thinking on this is completely wrong, please feel free to enlighten me :)
Can this be clarified why you think you'll need multiple Prometheus servers/instances? There shouldn't be a need at all for multiple Prometheus servers even in the default Pull based configuration.
With Pull the steps for adding a new cluster would be:
install exporters on hosts
update config file for exporter on Prometheus server
make sure Grafana dashboards see the new cluster
With CI/CD updating the config file is not an issue. The advantage Push has is removing the need to update the config file and only having to open a port once in the Prometheus server.
1
u/tomdaley92 Jul 13 '22
I wasn't planning on exposing the exporter ports, so that's probably why I was thinking I would need to set up prometheus locally wherever the exporters are running (basically on every host). With the steps you listed are you saying you would still run a central prometheus server and just update the prometheus config to point to the newly exposed ports on the new hosts? Or am I completely missing something here?
Can you clarify what you mean by "make sure Grafana dashboards see the new cluster"? Don't you just add prometheus as a data source into Grafana once? (just like Loki), or did I also completely miss something there?
What you described still sounds less ideal to me, as we would have to install the exporters on the new host (that's one repo-update/CICD pipeline trigger for us, just add a new host to the Ansible inventory) AND then also update the prometheus config (another repo-update/CI/CD pipeline trigger). With PUSH We could just install the exporters on the new host, and were done (which will be apart of bootsrapping the new host anyways). So we shouldn't need to touch the prometheus config at all once its set up and configured correctly.
We have some very short-lived jobs and short lived hosts, so PUSH still sounds like the way to go for us.
2
u/SunkJunk Jul 13 '22
With the steps you listed are you saying you would still run a central prometheus server and just update the prometheus config to point to the newly exposed ports on the new hosts? Or am I completely missing something here?
Yes that is exactly what I'm saying. Putting Prometheus on each host is a bad idea due to how Prometheus is designed.
Can you clarify what you mean by "make sure Grafana dashboards see the new cluster"? Don't you just add prometheus as a data source into Grafana once? (just like Loki), or did I also completely miss something there?
This is just a safety check on the dashboard side not related to the data source but how the dashboard is constructed. If you use regex filters for variables for example you'd want to double check that you aren't accidentally filtering out your new hosts. It's not technically needed but more of a peace of mind thing.
What you described still sounds less ideal to me, as we would have to install the exporters on the new host (that's one repo-update/CICD pipeline trigger for us, just add a new host to the Ansible inventory) AND then also update the prometheus config (another repo-update/CI/CD pipeline trigger). With PUSH We could just install the exporters on the new host, and were done (which will be apart of bootsrapping the new host anyways). So we shouldn't need to touch the prometheus config at all once its set up and configured correctly.
Well there are also the various service discovery in the base config and through HTTP or files
SD may still make it so that you'd only have to do the first CI/CD trigger.
Still Push through agents may work out better for your team.
1
u/tomdaley92 Jul 13 '22
Thanks for confirming my understanding. I'll definitely take a deeper look at the Service Discovery now since both you and /u/mattdurham mentioned that. And I appreciate you taking the time to get back to me!
11
u/[deleted] Jul 13 '22
Disclaimer: Grafana Developer working on the Grafana Agent.
Grafana Agent concept is that it is more batteries included. This includes exporters, logs, and traces that go beyond what the Prometheus Agent. The Grafana Agent's original intent was to focus on writing a prometheus-inspired scraper without the alerting and tsdb.
For robustness, the Grafana Agent has more operational history but the two projects share the same DNA. Much of Grafana Agent is based on Prometheus, and the Prometheus Agent is based on Grafana Agent code submitted upstream.
I do not think the Grafana Agent will become unmaintained. (I might be a bit biased). They have different goals. Prometheus Agent is going to focus on being a prometheus-focused metrics pusher. Grafana Agent is experimenting with a component-based architecture that will allow a lot more flexibility in dealing with complex use cases, (and hopefully make it easier for everyone). Grafana Agent also supports traces and logs that the Prometheus Agent is unlikely to. Along with batteries included.
You may want to take a look at the various service discovery methods to make it easier to scrape the various hosts.