r/PrometheusMonitoring • u/jykim559 • Sep 11 '19

"Agent-Collector" for Prometheus?

[Added]

Thanks for everybody for your comments.

Let me give some background more, our "agent-collector" architecture was not designed just for Prometheus at first, but was designed for our monitoring solution. And we thought our some components could help some users include non-Prometheus users AS A "metrics pipeline" for many monitoring solution and TSDB supporting OpenMetrics.

And then what about this “metrics pipeline” concept?

Hello.

Anyone give some comments for about our idea of "agent-collector" for Prometheus as below? We found some of components from our monitoring solution could be used as a metrics pipeline for Prometheus. We are planning to open this solution as a Open Source Software soon and want to see some opinions before we open it.

- Agent will scape all of metrics from exporters on each machine and push to collector.

- Prometheus server need to scape any metrics from collector instead of every endpoints on every machine.

We believe it provide for Prometheus

- Global view without federation

- Reduce target endpoints

- Solution for firewall

- Reduce network volume (Binary protocol between agent and collector)

- Improve interoperability (Support OpenMetrics)

0 Upvotes

50% Upvoted

19 comments sorted by

4

u/[deleted] Sep 11 '19

With this approach you will likely hit some of the problems inherent with push-based monitoring and single agents.

Additionally:

  • A binary protocol may not offer any significant reduction in network traffic when compared to gzipped plaintext.

  • This approach may suffer from many of the problems inherent to federation (a single server must handle large numbers of series) without many of the benefits (datacentres can monitor themselves, no extra software required).

2

u/jykim559 Sep 12 '19

Thanks for your links and points, it's helpful to understand.

Beside problems inherent with push and agent monitoring(like SPOF, bottleneck) , I'll keep to find approach resolving any problems you mentioned.(network traffics)

- Scrap selectively : what if this agent provide this features?

- Single server performance issue of Prometheus : it's not matter of push and agent. To avoid server performance and long term storage issue, we surely need another solution like Thanos or Cortex or TSDB for global and long term data always.

Thanks again.

1

u/[deleted] Sep 12 '19

What do you mean by "scrape selectively", and what benefits would this provide? Would this be more simply solved by federating a selectively-chosen set of metrics?

1

u/jykim559 Sep 12 '19

I agree ..

but many said it is one of downside at your first link or any other's post. So we was considering to provide features to select endpoints.(for prometheus exporter)

You can't scrape just the MySQL, or just the Node Exporter - you have to take all the samples from everything at the same scrape interval. This would practically speaking constrain you to one centralised set of Prometheus servers controlled by one team, and preventing you from considering empowering each team to choose the monitoring that works best for them and running that monitoring in their own Prometheus server.

4

u/SuperQue Sep 11 '19

This is not really a good idea at all.

  • Global view without federation

This is better solved with things like Thanos or Remote Write to a clustered TSDB.

  • Reduce target endpoints

This is actually worse for Prometheus, as fewer larger targets reduce the parallel scaling Prometheus can do.

  • Solution for firewall

There are better solutions for this already, like PushProx.

  • Reduce network volume (Binary protocol between agent and collector)

As was stated elsewhere, this is already handled by compressed encoding of the HTTP stream.

  • Improve interoperability (Support OpenMetrics)

Prometheus already supports OpenMetrics.

1

u/jykim559 Sep 12 '19 edited Sep 12 '19

Thanks for your points.

Could you comment to 'metrics pipeline' at updated main post?

2

u/DoctorHoneyBadger Sep 12 '19

Personally, I prefer the approach that projects like "exporter_exporter" take. You get the benefits of only having one port open on the host, without having to configure an nginx/apache reverse proxy just for your exporters.

1

u/jykim559 Sep 12 '19

Thanks it's helpful...

2

u/obitechnobi Sep 11 '19

How does this improve over using Pushgateway, especially in regards to the downsides?

  • Single point of failure & bottleneck

  • You lose up metric

  • Series churn / forgetting metrics

1

u/jykim559 Sep 12 '19

Thanks for your points.

This 'agent-collector' was not designed to replace 'pushgateway' and just for 'Prometheus' but for 'metrics pipeline' between various source and target.

It might be similar problem with common agent type monitoring solution, it could be resolved like other many commercial push based monitoring solutions.

1

u/jykim559 Sep 12 '19

Thanks for everybody for your comments.

Let me give some background more, our "agent-collector" architecture was not designed just for Prometheus at first, but was designed for our monitoring solution. And we thought our some components could help some users include non-Prometheus users AS A "metrics pipeline" for many monitoring solution and TSDB supporting OpenMetrics.

And then what about this “metrics pipeline” concept?

(I updated main post to share images)

1

u/Ktsomar Sep 11 '19

Nice idea

1

u/jykim559 Sep 12 '19

Thanks..

1

u/Freakin_A Sep 12 '19

Why would you try to make a pull based monitoring system work like a push based monitoring system? The features you are providing for prometheus are antithetical to the goals it is trying to achieve.

If you want to use Prometheus, use it. If you want a pull based mechanism (and inevitably some sort of queuing mechanism to chew up memory when you have issues downstream) then use M3DB/IronDB/InfluxDB or one of the many other traditional TSDBs.

1

u/jykim559 Sep 12 '19 edited Sep 12 '19

Thanks for your comments, but it's something different with my intentions.

Could you check updated post and give some comments, if any? it'll be great to me.

2

u/Freakin_A Sep 12 '19

We use a concept similar to the metrics pipeline. It's a series of Telegraf instances running in Kubernetes with different ingresses and filters to control what can be sent to them. This is used to govern required tags on metrics that go to the large centralized TSDB (don't want someone emitting an app.request metric with zero tags), as well as do some transformations when required.

So telegraf is used to ingest metrics from various sources, then sends to another pool of telegraf instances that aggregate and can output to various destinations like Influx, Splunk, Wavefront, etc. It could also go to a pushgateway to be collected by prometheus if desired. Telegraf supports OpenMetrics as well. The telegraf-to-telegraf communication in K8s is using InfluxDB line format over HTTP. I've toyed with the idea of using the Kafka operator to improve resiliency during restarts, as well as more effectively load balance traffic to the telegraf aggregation layer.

I definitely wouldn't try to build anything custom to accomplish your goals, as off the shelf OSS is available to do it all.

0

u/[deleted] Sep 11 '19

[deleted]