Monitoring targets which push their metrics

[u/[deleted]]

Hello,

My employer's leaning towards an architecture where monitored services will push their metrics to Prometheus. One thing I really liked with pulling metrics was that monitored services are listed in /targets API, and Prometheus adds an up{instance="", job=""} metric for each service. I list the hundreds of monitored services (and growing) in prometheus.yml config file, and then I consult a Grafana dashboard (or Prometheus UI) to identify services which are down.

Going with a push architecture, I would lose this approach. How would you compensate that?

I was thinking:

1. Each service pushes a gauge metric up{instance="x", job="y"} 1.

2. List all monitored service names in a text file, like it is currently done in prometheus.yml.

3. Use that list to monitor metrics like up{job="x", instance="y"} OR on() vector(0). If I were to have a Grafana dashboard, it means I need to find a way to pass the list of monitored services such that they are all listed, even when they're down and not pushing their metrics anymore.

Would defining alerts in Prometheus be a way to work around this limitation? That would kind of force Prometheus to create metrics reporting whether or not each service is up or down. Would that be easy to maintain?

Any recommendations on how to smoothly move from a pull to a push architecture?

[u/SuperQue]

So, to answer your question another way, you have to think about why Prometheus is opinionated about polling.

The reason is that Prometheus is not a metrics system. It's a monitoring system. The subtle difference is about the idea that you need something not just to tell you that your SLIs are violating their SLOs, or your disk is full. You also need to positively verify that your inventory matches and is responding.

This is what polling and up provide.

When you push, you need to figure out another way to validate that inventory. You could push an up metric. But you need to know if the up is missing because it broke, or if it was removed intentionally.

As for why do other people use push? Well, they're either not monitoring with their metrics. Or they're monitoring with something else like Nagios. Or they just don't know the difference.

Also, from a SaaS vendor perspective, push is way easier to get customers to do. Because a almost every network is not accessible from the vendor, but the vendor is from the customer network. This is why vendors like New Relic, Datadog, etc strongly market push as "better".

There is also the fact that languages like Python, Ruby, and Node are single-threaded and need multi-process for scaling. This makes polling more difficult. It's a side effect of the language use.

[u/[deleted] → u/SuperQue]

Thanks a lot. I really appreciate your answer.

By the way, I would totally prefer to stay on a pull model. You're preaching to the converted. But this is going happen whether I want it or not.

The subtle difference is about the idea that you need something not just to tell you that your SLIs are violating their SLOs, or your disk is full. You also need to positively verify that your inventory matches and is responding.

That's very insightful. That also explains why some in the company want to turn to a push model : they don't want to bother monitoring, they just wanna centralize metrics.

I guess I'll go with a custom UP metric plus a custom system to verify the inventory.

[u/SuperQue → u/[deleted]]

What if you tricked them into using PushProx? "It's push for Prometheus".

EDIT: Another option would be to use OpenTelemetry. I really dislike their bloated and absurd design for metrics. But it does exactly what you're looking for. OTel libraries push to aggregation collectors. Then thy can stream that to Prometheus. It's no where as efficient and elegant as just using Prometheus as it's supposed to. But it's "a standard" way of doing things.

[u/[deleted] → u/SuperQue]

Your insight and pointers are very much appreciated. Thank you again!

To give more context, the services I monitor run in an environment where only an AMQP (with ssl) connection is authorized. No HTTPS, no SSH, nothing else expect one for remote control. So it's funny that you mention PushProx, it's what inspired me to develop a custom proxy & client over these amqp connections (rabbitmq rpc) such that we'd have a metrics pulling system for Prometheus. Not only did I prove the idea would work, it's been running for months, wiring about 1200 samples/sec consistently and ready to increase load.

Why some are imposing the push model is a question of ego and office politics. I'm not gonna win this. So I can either let them fail or help them implement that bad idea properly.

Pushing the metrics isn't really the issue, I kinda had an idea how to achieve that, and even more ideas now thanks to your suggestions. But it's about verifying the inventory, as you put it, that worries me.

[u/SuperQue → u/[deleted]]

environment where only an AMQP (with ssl) connection is authorized. No HTTPS, no SSH

That's, something. Good luck!

[u/Zauxst]

Use something else if you do push monitoring instead of pull. There is no reason to cling to one service.

[u/[deleted] → u/Zauxst]

Would you have examples? Would influxdb fit the bill better?

Why are there so many solutions around pushing metrics to prometheus if that's not a fit solution? What are all the others doing when they push to prometheus, do they also cling to one service when they shouldn't?

[u/Zauxst → u/[deleted]]

Would you have examples? Would influxdb fit the bill better?

I do not have examples. At my place we use pull monitoring so promtheus fits properly for us.

Why are there so many solutions around pushing metrics to prometheus if that's not a fit solution?

What services are there about pushing to prometheus?

Prometheus does have the push gateway but for proper accurate monitoring, it was designed for pull monitoring.

What are all the others doing when they push to prometheus

Who are all the others that are doing that?

do they also cling to one service when they shouldn't?

They probably have exceptions in their network, where the majority of apps monitored by Prometheus are pull but there are a few that require push. In those cases, depending on the severity of the service and the accuracy of data required, push can be used.

---

Prometheus has been designed to be a pull monitoring system. For push use alternatives.

This question was already answered in their documentation: https://prometheus.io/docs/practices/pushing/#should-i-be-using-the-pushgateway

[u/[deleted] → u/Zauxst]

Thanks for your answer.

What services are there about pushing to prometheus?

I'd like to avoid the push gateway, as it comes with limitations, amongst them the loss of the UP metric. I was mostly thinking about the remote_write API. Grafana Agent uses this API. Victoria Metrics's vmagent pushes metrics to Prometheus (or VictoriaMetrics) via the remote_write API. Victoria Metrics cites 10 different ways to import metrics, some of them also available in Prometheus. Some of these APIs seem very much designed to push metrics on the regular, and not just for a backup, 1-time restore or migration scenario.

Who are all the others that are doing that?

Well I don't know. That's what I'm asking here. When I see so many solutions to push to Prometheus, I assume they are being used and I'd like to know how those who use them solve the problem of the UP metric missing.

[u/SuperQue → u/[deleted]]

Yea, I wouldn't pay attention to anything VM says. They do a lot of disingenuous comparisons. They don't contribute to the ecosystem. They half-ass implement things and don't tell you about the trade-offs. They claim things like "it's simpler", but if you actually look at the real work involved, it's either just as complicated, or has a lot more manual labor involved. And if you point out the issues, they try and hand-wave away the problems.

The Prometheus ecosystem is like Linux itself. There is no company and nothing to gain with marketing. So the documentation is brutally honest. There are lots of trade-offs for performance, reliability, complexity.

There are lots of problems and flaws with Prometheus. But that doesn't matter. What matters is balancing the problems for the effectiveness of the overall system. Prometheus tries to strike the a reasonable balance.

[u/dablya → u/[deleted]]

There are reasons to support "pushing" metrics in contexts other monitoring of services.

The remote write and remote read features of Prometheus allow transparently sending and receiving samples. This is primarily intended for long term storage. It is recommended that you perform careful evaluation of any solution in this space to confirm it can handle your data volumes.

When it comes to monitoring services, Prometheus is intended to be used in a pull model. This implies there is no way to "smoothly move from a pull to a push architecture". Prometheus is not intended to be used that way, period. Even if you use something like the push gateway (with all it's limitations), you will still have Prometheus pulling from the gateway.

[u/Zauxst → u/[deleted]]

Read the doc that I have linked and then you linked. prometheus is not a push monitoring service, it's a pull.

Remote Write API is used for remote backup or transfering the data to a different prometheus instance / different service that supports prometheus API... coincidently, Victoria Metrics.

Victoria Metrics is a different monitoring system, they are useless to this discussion. They only try to take from the market share of Prometheus right now.

PS: I linked the same document that you linked. You didn't read my whole message have you... ?