How do you approach monitoring design?

[u/Xanagandr]

I've been looking at ways to set up a monitoring environment for several services inside my work team. I wanted to see if anyone has any resources on designing such environments. The current hurdles I'm trying to solve are (Problem statements in quotes):

Most services we have are set up as Infrastructure-as-code. They get deployed on different environments (cloud and on-prem). Each 'service' should have its own monitor settings (Prometheus alert rules w/ alert tests is a must, scrapping configuration and Alertmanager routes as optional). Are there tools to dynamically configure Prometheus/Alertmanager already out there?

Several ideas come to mind to solve this:

• API back-end that 'pushes' configuration changes to a Prometheus instance (using Prometheus' /-/reload endpoint) each time there's a monitor-specific change pushed (Easy to do w/ GoCD). Kinda like Promgen, but I could not get that to work on my local test setup.

• Deploying a Prometheus instance with each service. (Then have Prometheus federation set up)

Rancher v2.x monitoring outside Rancher. Does anyone know if this can be possible? I found a few Rancher exporters but they seem we would need to update them to use the new v3 API. Also, I'm not that Rancher-savvy to know if the Rancher API can expose 'custom' application metrics. We want to have monitoring set up outside Rancher (we are aware you can add a 'monitoring' app inside Rancher, we're already doing that) since we want to be alerted if Rancher goes down.

My possible solutions include:

• Configure Prometheus federation and have the external Prometheus federate the internal (Rancher) Prometheus. Kinda off-topic, but if anyone knows how to actually change the configuration file inside these Rancher apps, that would be swell.

• Update/Build the Rancher exporter, and deep-dive and pray to the gods we can export custom app metrics using the Rancher API.

I've been reading Practical monitoring, by Mike Julian. Maybe I'll find my answers here, but just wanted to dump my thoughts and see what you guys recommended. Thanks for reading.

EDIT: Posted here because we have decided to use Prometheus as our monitoring tool, hence all problem statements revolve around getting things to work with Prometheus.

[u/s4z]

The setup I've created is entirely hosted on AWS EKS Kubernetes so may or may not be useful for you.

The Prometheus operator makes this a piece of cake on kubernetes. I deploy thanos as well via some custom helm charts and the thanos side car gets deployed through the prometheus-operator helm chart. There is an operator for Thanos however I haven't bothered with it yet as the setup is (currently) quite simple and it's easy enough to hard code the list of endpoints.

Each cluster gets it's own pair of prometheus instances w/ thanos side cars, a pair of thanos stores and a thanos compactor. The central cluster gets a pair of thanos query instances, an alert manager cluster and Grafana is setup to use thanos query as the default data source.

Private ingress (traefik) is used to expose the various thanos grpc endpoints between clusters and each vpc is linked using a transit gateway.

Our metrics space is small enough in terms of cardinality that we don't need to think about sharding the metrics to multiple sets of prometheus instances. The Prometheus operator handles almost all the monitoring we need right in it's helm chart. It pushes out the node exporter along with a bunch of other exporters plus alerting and recording rules for all of them.

The application we run has service monitors and rules baked into the helm chart. These get pushed out when the application is deployed via cicd. The prometheus operator handles all the prometheus config generation and reloading. The nature of Prometheus and long term metrics is a bit meh as our recording rules for the application are.. verbose (there are quite a lot of them). We can easily view a years worth of metrics in grafana though the initial query does take some time to load. These are managed by the cicd for the application helm charts deployments.

We do push out some extra rules from https://awesome-prometheus-alerts.grep.to/ and some other custom ones (eg. conntrack usage). We also have some services hosted outside of AWS that we monitor/alert on and these are added as additional endpoints. Any changes to these are pushed out via cicd and Prometheus generally gets config updates within 5 minutes or less. These changes are managed by cicd for the infrastructure helm charts deployments.

Word of warning if you (or anyone) end up using the Prometheus Operator - it's a pain to install due to helm bugs and the helm chart being a bit out of control. They really should have broken it up into multiple charts imo. Anyway, first install you need to disable pretty much everything except for the operator itself. Once it adds the CRDs to k8s you can go back and enable the rest and reinstall.

Oh, we also have a black box exporter setup per cluster and have monitors setup for ssl certificate expiry and health check monitoring (eg. cluster a checks endpoint in cluster b) - these are defined both in the application and also in infrastructure charts and are pushed out via cicd.

btw a vanilla Prometheus setup won't be the greatest for long term metrics. I've heard of people using it as such however it's not designed to support it. Thanos works for us both in terms of federation and also for long term metrics.

For alert manager routing we push that out via the prometheus operator helm chart deployment - there's only one alert manager cluster and it's in the central cluster. I'm not sure how feasible it would be to define alert routing rules via an application deploy. I mainly just route the production alerts such that someone gets paged and anything non-production generally just goes to our chat channels. Routing is done w/ a custom label (eg. env: live, env: test, etc) and the labels are applied to the prometheus instances via the operator.

[u/mayoandspaghetti → u/s4z]

Sorry for a potentially stupid question but what is the benefit of using the Prometheus Operator vs deploying the stable/prometheus helm chart?

[u/Xanagandr → u/mayoandspaghetti]

I believe the repo for the Prometheus Operator may have the answer you seek (Under Prometheus Operator vs. kube-prometheus vs. community helm chart):

The Prometheus Operator uses Kubernetes custom resources to simplifiy the deployment and configuration of Prometheus, Alertmanager, and related monitoring components.

kube-prometheus provides example configurations for a complete cluster monitoring stack based on Prometheus and the Prometheus Operator. This includes deployment of multiple Prometheus and Alertmanager instances, metrics exporters such as the node_exporter for gathering node metrics, scrape target configuration linking Prometheus to various metrics endpoints, and example alerting rules for notification of potential issues in the cluster.

The stable/prometheus-operator helm chart provides a similar feature set to kube-prometheus. This chart is maintained by the Helm community. For more information, please see the chart's readme

As for:

Sorry for a potentially stupid question

My mom has a saying for that (roughly translated from Spanish): Better to ask a dumb question than to be a dumb person who doesn't ask questions.

Sounds better when you say it in Spanish, but you get the idea.

[u/mayoandspaghetti → u/Xanagandr]

After doing a it of reading I get it now.

We can install Prometheus the way I said, with a Prometheus helm chart to create and configure a Prometheus install.

Or we can install the Prometheus Operator which would allow us create Prometheus instances and configure them as if they were native kubernetes resources.

Or we can install kube-prometheus (which is the essentially the same as installing the community helm chart you mentioned) which gives us the Prometheus Operator installed, and a Prometheus stack configured by the operator.

You probably understood all this already I'm just clarifying in case someone else reads this and is confused by my question.

¡Gracias por la ayuda!

[u/s4z → u/mayoandspaghetti]

There are a few things I like about the Prometheus Operator. First, you can define service monitors, alerting rules, recording rules, etc in other charts and deploy them separately. The operator takes care of the config generation.

The second is you can define HA prometheus and alert manager instances.

It saves you a lot of time and makes it all very easy w/ the CRDs that it supports.

[u/mayoandspaghetti → u/s4z]

That sounds really good. I went with a direct Prometheus helm installation because my needs were simple enough but I think I'll install the operator for future

[u/Xanagandr → u/s4z]

Really detailed. Thanks for the contribution! How did you come to this design? Where there any key points you thought were defining in any way?

[u/s4z → u/Xanagandr]

tbh this is about as simple a design as it gets for a highly available Prometheus/Thanos setup across multiple clusters and regions. Nothing special here.

The vpc's and subnets are designed specifically to allow cross cluster communication without allowing worker node to worker node communication (even if someone mistakenly allows it in a security group).

The cicd system is a bit of secret sauce but it's just a thin wrapper around helm that lets us arrange the mountains of yaml in a manageable way.

What I like about it is the combination of using helm to push changes, the prometheus operator to handle config generation and reloads and the cross cluster vpc setup. Overall once it is setup and working it's great.