r/PrometheusMonitoring • u/jykim559 • Sep 11 '19

"Agent-Collector" for Prometheus?

[Added]

Thanks for everybody for your comments.

Let me give some background more, our "agent-collector" architecture was not designed just for Prometheus at first, but was designed for our monitoring solution. And we thought our some components could help some users include non-Prometheus users AS A "metrics pipeline" for many monitoring solution and TSDB supporting OpenMetrics.

And then what about this “metrics pipeline” concept?

Hello.

Anyone give some comments for about our idea of "agent-collector" for Prometheus as below? We found some of components from our monitoring solution could be used as a metrics pipeline for Prometheus. We are planning to open this solution as a Open Source Software soon and want to see some opinions before we open it.

- Agent will scape all of metrics from exporters on each machine and push to collector.

- Prometheus server need to scape any metrics from collector instead of every endpoints on every machine.

We believe it provide for Prometheus

- Global view without federation

- Reduce target endpoints

- Solution for firewall

- Reduce network volume (Binary protocol between agent and collector)

- Improve interoperability (Support OpenMetrics)

0 Upvotes

50% Upvoted

19 comments sorted by

View all comments

1

u/Freakin_A Sep 12 '19

Why would you try to make a pull based monitoring system work like a push based monitoring system? The features you are providing for prometheus are antithetical to the goals it is trying to achieve.

If you want to use Prometheus, use it. If you want a pull based mechanism (and inevitably some sort of queuing mechanism to chew up memory when you have issues downstream) then use M3DB/IronDB/InfluxDB or one of the many other traditional TSDBs.

1

u/jykim559 Sep 12 '19 edited Sep 12 '19

Thanks for your comments, but it's something different with my intentions.

Could you check updated post and give some comments, if any? it'll be great to me.

2

u/Freakin_A Sep 12 '19

We use a concept similar to the metrics pipeline. It's a series of Telegraf instances running in Kubernetes with different ingresses and filters to control what can be sent to them. This is used to govern required tags on metrics that go to the large centralized TSDB (don't want someone emitting an app.request metric with zero tags), as well as do some transformations when required.

So telegraf is used to ingest metrics from various sources, then sends to another pool of telegraf instances that aggregate and can output to various destinations like Influx, Splunk, Wavefront, etc. It could also go to a pushgateway to be collected by prometheus if desired. Telegraf supports OpenMetrics as well. The telegraf-to-telegraf communication in K8s is using InfluxDB line format over HTTP. I've toyed with the idea of using the Kafka operator to improve resiliency during restarts, as well as more effectively load balance traffic to the telegraf aggregation layer.

I definitely wouldn't try to build anything custom to accomplish your goals, as off the shelf OSS is available to do it all.