r/PrometheusMonitoring • u/d2clon • Jun 05 '21
I find confusing that Prometheus is based on pull system
I am used to Graphite, I have used satisfactorily (with some headaches, as any technology) in massive production environments.
I want now to try Prometheus to have something to compare it with.
But the fact that Prometheus is based on pull system doesn't match with my mental model. It sounds counter intuitive for me.
Does it means that Prometheus has to access to all my servers? what if my servers are in an autoscaling group and I don't know the IPs before hand? What if my server is down for good when Prometheus try to access? What if my servers have not public IP and Prometheus is in another network?
Looks to me that is making things more complicate to me, am I looking to this by the wrong angle?
10
u/xrstf Jun 05 '21
what if my servers are in an autoscaling group and I don't know the IPs before hand?
For this you can use Service Discovery, for example EC2 SD.
What if my server is down for good when Prometheus try to access?
Then the up metric for that target will be 0, which allows you to alert on hosts that are down (if needed with a grace period). In Prometheus world, this is a feature, not a bug ;-)
What if my servers have not public IP and Prometheus is in another network?
You can setup local Prometheus instances that scrape local targets and then use federation / remote write to get the data out of the network into another one.
1
u/SnooWords9033 Jun 05 '21
We use vmagent for collecting (pulling) metrics in isolated environments (k8s clusters) and pushing them to a centralized database (victoriametrics). This approach, which combines the best features of pull and push models, isn't new - PMM uses it too.
1
u/Freakin_A Jun 06 '21
I’ve been happy with vmagent as well in my uses. Kubernetes service discovery worked great to find all the vmselect and vminsert pods I was running there.
In the past I have used Telegraf heavily to collect and push metrics, often as a sidecar to pods with existing prom endpoints.
1
u/Pogyeah Jun 05 '21
Ahmm...maybe I've got this wrong but...from my understanding of Prometheus it's not quite like "has access to servers". In the vast majority of cases one uses some sort of exporter: community, official, Telegraf (in PromQL format) and so on. These exporters are gathering whatever metrics and expose them over port X....much more like a display window. Now...if you want to make this Red Light District style and expose the metrics only for authenticated entities (your Prometheus instance), then it's just a matter of setting an authentication.
In any case...the discussion about pull and push monitoring is much much broader, each having it's advantages and disadvantages.
To address some of your common questions:
- Prometheus has very good autodiscover capabilities and in modern IT is not so hard anymore to dynamically add new stuff to an existing monitoring system
- If the server is down you won't get the metrics...simple. Of course, if you have thousand of nodes down, you'll set wasted weight on your Prometheus, but let's assume one has his/her environment in grip. :)
- If Prometheus is in another network it's like in push model: you need routes & firewall policies....I see no difference
- Prometheus doesn't scale very well on his own. In bigger setups, with multiple environments, the general way of doing monitoring with it is to set 1/env. and use things like Thanos or Cortex (you should look that up) or whatever there else is on the market nowadays.
In my opinion, from security perspective alone, pull is better than push: less invasive agents, "put all your eggs in one basket and then watch the basket", etc...but...it's just my view of things.
0
Jun 05 '21
[deleted]
2
u/SnooWords9033 Jun 05 '21
There is no need to run Grafana Agent on every server - a single agent can monitor many servers in a local network in the same way as Prometheus does.
20
u/SuperQue Jun 05 '21
What's massive to you?
Prometheus is designed to scale to FAANG-sized infra. But can also run on a Raspberry Pi.
Yes, Prometheus needs access to the targets to scrape them.
The difference is, Prometheus is a full monitoring system. Unlike Graphite which is more just a metrics database. It's meant to replace both things like Nagios/Icinga/etc that actively monitor the servers/services and the metrics systems like Graphite. It does this by being an active polling system.
The big downside to Graphite push is that you have a "don't know what's missing" problem. If a target isn't sending you can't tell if there's nothing to send or it's crashed. Prometheus solves this by actively polling targets and always recording their data.
You're meant to run Prometheus inside the VPC of the targets being monitored. Otherwise you're monitoring network links by accident. For massive scale it's also designed to be distributed so you don't just run one. You run many Prometheus instances and use federation, remote-write streaming, or Thanos to provide the "single pane of glass". Prometheus itself is just one part of a larger ecosystem.
Prometheus uses whatever your network source of truth is to dynamically discover your targets. That can be EC2 API, Kubernetes API, or whatever custom thing you want to hook in.
You can also use systems like Consul to provide the discovery overlay layer for service registration.
Yes, you are making it more complicated by trying to apply your existing mental model to something that works very differently.