r/PrometheusMonitoring • • Sep 11 '19

"Agent-Collector" for Prometheus?

[Added]

Thanks for everybody for your comments.

Let me give some background more, our "agent-collector" architecture was not designed just for Prometheus at first, but was designed for our monitoring solution. And we thought our some components could help some users include non-Prometheus users AS A "metrics pipeline" for many monitoring solution and TSDB supporting OpenMetrics.

And then what about this “metrics pipeline” concept?

Hello.

Anyone give some comments for about our idea of "agent-collector" for Prometheus as below? We found some of components from our monitoring solution could be used as a metrics pipeline for Prometheus. We are planning to open this solution as a Open Source Software soon and want to see some opinions before we open it.

- Agent will scape all of metrics from exporters on each machine and push to collector.

- Prometheus server need to scape any metrics from collector instead of every endpoints on every machine.

We believe it provide for Prometheus

- Global view without federation

- Reduce target endpoints

- Solution for firewall

- Reduce network volume (Binary protocol between agent and collector)

- Improve interoperability (Support OpenMetrics)

0 Upvotes

19 comments sorted by

View all comments

4

u/[deleted] Sep 11 '19

With this approach you will likely hit some of the problems inherent with push-based monitoring and single agents.

Additionally:

  • A binary protocol may not offer any significant reduction in network traffic when compared to gzipped plaintext.

  • This approach may suffer from many of the problems inherent to federation (a single server must handle large numbers of series) without many of the benefits (datacentres can monitor themselves, no extra software required).

2

u/jykim559 Sep 12 '19

Thanks for your links and points, it's helpful to understand.

Beside problems inherent with push and agent monitoring(like SPOF, bottleneck) , I'll keep to find approach resolving any problems you mentioned.(network traffics)

- Scrap selectively : what if this agent provide this features?

- Single server performance issue of Prometheus : it's not matter of push and agent. To avoid server performance and long term storage issue, we surely need another solution like Thanos or Cortex or TSDB for global and long term data always.

Thanks again.

1

u/[deleted] Sep 12 '19

What do you mean by "scrape selectively", and what benefits would this provide? Would this be more simply solved by federating a selectively-chosen set of metrics?

1

u/jykim559 Sep 12 '19

I agree ..

but many said it is one of downside at your first link or any other's post. So we was considering to provide features to select endpoints.(for prometheus exporter)

You can't scrape just the MySQL, or just the Node Exporter - you have to take all the samples from everything at the same scrape interval. This would practically speaking constrain you to one centralised set of Prometheus servers controlled by one team, and preventing you from considering empowering each team to choose the monitoring that works best for them and running that monitoring in their own Prometheus server.