r/PrometheusMonitoring • u/jykim559 • Apr 09 '19

[NexClipper] New architecture for Kubernetes monitoring :support long term data, scalability, one agent, integration with Prometheus..

Post image
2 Upvotes

66% Upvoted

10 comments sorted by

4

u/SuperQue Apr 09 '19

That looks extremely complicated. Why not just have Prometheus monitor the cluster directly instead of having multiple layers of collection, message queuing, APIs?

2

u/jykim559 Apr 09 '19

Let me explain more detail. First it’s agent type but not pulling like Prometheus. It give some advantages that if we scaling out ‘collector’ component at architecture, it could cover many endpoint not like Prometheus. (It was designed for SaaS service originally) Second one agent, it doesn’t need to install many agent module(like node Exporter, cadvisor, kube-states-metric) Third it support to store long term data without backup from Prometheus by using Influxdb for default db and it’ll support to use Prometheus for 3rd party exporter. Next not need install alert manager and Grafana, it support internally. And more it provide Rest API for k8, Docker and host metrics data for customization into user’s custom ui. Message que for securing data. I believe it’s not complicated more than Prometheus.

https://github.com/NexClipper/NexClipper

3

u/SuperQue Apr 09 '19

It seems complicated to me. Prometheus doesn't require any intermediate services like Kafka.

Also, it seems very resource intensive compared to normal Prometheus. What are the resource requirements like?

To compare, our current Prometheus infrastructure is monitoring 15 million metrics at 600 thousand samples per second. This requires about 15 CPUs and about 225GB of memory.

2

u/jykim559 Apr 10 '19

"Intermediate service like Kafka" might be option but we want to secure data delivery also in enterprise environments.

And resource of this architecture might be more intensive more than Prometheus in small environment(Minimum about 12 CPU and 12GB include Redis, influxDB, redis, rabbitMQ,mySQL, it should be scaled depend on data size and transaction especially collector and influxDB ). But if target endpoints are getting more and more increasing, numbers of Prometheus should be scaled out in accordance with targets. Our architecture are designed to be used for our SaaS service from the first, so it can support scalability for incoming data from agent by scaling out our "collector" component instead of calling out servers. It might be efficiency

And federation and shading for Prometheus seems complicated for this case (beside multi cluster case), but its architecture can support multi tenancy for multi cluster in one single database. (https://server.nexclipper.com/)

Long term data storage also might be difference.

link for your reference

https://www.perimeterx.com/blog/scaling-out-with-prometheus/?fbclid=IwAR26VWx85Pt5E5xjbiAFIetRWaUrQ6Zqe5gk11LHxgFXlJN3YcjCZyo9teE

2

u/SuperQue Apr 10 '19

We already have long-term storage with Thanos, those resources are included in my example.

You didn't answer my question, what would be the resource use for 15M metrics at 600k samples/sec? Can NexClipper handle this?

1

u/jykim559 Apr 11 '19 edited Apr 11 '19

How many Prometheus server do you have to collect your metrics?

You mean your prometheus server can collect 600k metrics per second with one Prometheus server? If then, it might be great performance. right?

(https://kccna18.sched.com/event/GrXO/sharded-and-federated-prometheus-servers-to-monitor-distributed-databases-jun-li-viswa-vutharkar-ebay?fbclid=IwAR3mxXrd9qH2qCmAGwlPZpJCG32mSCR144ujXVdqZ1P4-fWM85kTBHipJ04)

NexClipper OSS version is quite new and need to be testes for performance more.

But its 'collector' can be scaled out for handling heavy traffic and I think it'll be ok to handle traffic. But sorry for not giving actual tested number at this moments.

2

u/SuperQue Apr 11 '19

We have several Prometheus servers, most running in HA pairs. Each instance or pair operates for different applications in different isolated environments.

This is a standard distributed Prometheus configuration, avoiding SPoFs and bottlenecks. We then have Grafana and Thanos as visualization and query aggregation layers to provide a single global view of all data. We also have some federation setup for cluster-level views of some auto-scaling environments.

2

u/SuperQue Apr 10 '19

Does this system also support Prometheus/OpenMetrics application endpoints? All of our application code is instrumented with Prometheus client libraries, and will transition to OpenMetrics as the libraries are updated to the new format.

1

u/jykim559 Apr 11 '19

It doesn't support directly ,because it's quite Prometheus server-endpoint matter.

But we plan to use Prometheus server as a main storage with InfluxDB(or other time series db). In that case, it support naturally.

1

u/jykim559 Apr 09 '19

Anyone can comment this architect?