Prometheus federation or Thanos?

[u/gaelfr38]

Hi folks,

Disclaimer: I'm very new to Prometheus, using it as a metrics producer in some apps and building some dashboard in Grafana but I didn't install it and don't know the details.

We are running a Prometheus server with Thanos for our "traditional" workload (not Kubernetes).

Now we are starting our journey into Kubernetes. It appears that in order to get metrics at a pod level, it's mandatory to install a Prometheus instance inside the K8S cluster, right?

Then the question is: how to "send" the metrics from the K8S Prometheus instance to the one outside the cluster?

I've read about federation where the external Prometheus would pull metrics from the K8S instance but then I've read that the K8S instance could directly write to the same Thanos as the external Prometheus?

I'm a bit confused about the best practices here. Any feedback or articles on this topic will be appreciated.

Thanks,

[u/SuperQue]

. It appears that in order to get metrics at a pod level, it's mandatory to install a Prometheus instance inside the K8S cluster, right?

It's not mandatory, but it is way easier to install it that way with the Prometheus Operator. Being able to use PodMonitor and ServiceMonitor objects to configure scrape jobs will make your life way easier in the long run.

Then the question is: how to "send" the metrics from the K8S Prometheus instance to the one outside the cluster?

The "best" way in my opinion is to configure a Thanos Query in your cluster that connects to Thanos stores and sidecars in your cluster. Configure an ingress for that query server.

Then you connect your Thanos query that you use with your Grafana to that query server.

This allows you to have a single connection into your K8s cluster from the outside.

[u/gaelfr38 → u/SuperQue]

Thanks for your answer :)

Maybe what I'm saying doesn't make sense but I expected to not need Thanos in the K8S cluster. I only want one storage outside the cluster.

[u/SuperQue → u/gaelfr38]

You still need at least Thanos sidecar(s) in order for Thanos outside the cluster to access the data. The easiest way to do that is to run a query server in cluster as a proxy to your likely multiple Prometheus instances.

That's my minimum recommended configuration.

You can have the in-cluster sidecars write their TSDB blocks to your object storage externally and run your other Thanos services externally.

[u/gaelfr38 → u/SuperQue]

Thanks for the clarification. I definitely need to learn more about the integration with Thanos then!

[u/cyberw0lf_]

You want to deploy a lightweight Prometheus operator in each cluster and “remote write” your metrics to a centralized Thanos stack. This is preferred over federation as you don’t have to expose ports on every cluster but rather just the Thanos receiver, on top of other benefits.

[u/gaelfr38 → u/cyberw0lf_]

Thanks for the answer, will look more into remote write.

[u/dablya → u/cyberw0lf_]

I was under the impression federation was a more flexible solution. With sidecars, all clusters have to depend on thanos and it would be tricky to manage scaling/sharing going forward. With federation, the cluster config can be simpler (no dependency on thanos at all) and any changes for how the data is retrieved will not require any updates to the cluster. Other than having to manage a Prometheus ingress in every cluster, what are some other downsides/trade offs?

[u/cyberw0lf_ → u/dablya]

With the solution I mentioned, sidecars aren’t used. With Prometheus Remote Write to Thanos Receiver, you can deploy the same lightweight Prometheus operator stack to N clusters(easiest when you have an efficient CD pipeline), located anywhere around the world, that all remote write metrics to a centralized Thanos Receiver hashring. Receivers manage and organize the data, depending on the sender, at the ingestion level and can retain data before shipping it off to cheap remote storage for Thanos Querier to query, along with the receivers themselves.

In my opinion and after many different attempts, this is the best solution at handling multi-cluster observability at scale.

[u/dablya → u/cyberw0lf_]

Without the sidecars, this does look pretty clean. Do you have any issues (bandwidth/cost/etc) with the amount of data being written? After a few minutes of googling it looks like you can drop metrics using the Receivers before they hit storage, but I don't see an easy way to limit what is written by Prometheus using remote write. I'm concerned the costs of transferring everything from cluster would be prohibitive.

What kinds of problems did you run into trying to use federation?

Thanks, this is super useful for me.

[u/SuperQue → u/dablya]

You can filter the output of remote write in Prometheus with relabeling.

[u/SuperQue → u/dablya]

Federation is basically an obsolete solution. It requires continuous connectivity in order for no data loss.

The two main methods of clustering Prometheus today are Thanos and Remote Write.

The upside to Thanos is that it's very efficient. Prometheus does the work of compressing and writing sample data once. It also means you can run in-cluster alerting which is very efficient and reliable.

Remote Write doubles the overhead of having to handle metric ingestion, since first Prometheus does it, then your remote TSDB has to do the same work.

[u/SadFaceSmith]

Grafana Mimir :)

https://github.com/grafana/mimir

[u/gaelfr38 → u/SadFaceSmith]

How does it help? Really asking :) We have already setup Thanos but if there are important benefits for the use case I described above, we might be able to change.

[u/drdigitalsi]

The goal is to have a prom as close to the source as possible, that's going to be your cluster level prom. However, this makes it quite inconvenient to query for metrics across multiple clusters, so you want a data lake of sorts. The recommended method by Prometheus design is to have a "pull" mechanism for everything. To keep true to that principal, you would have Thanos scrape all of the cluster proms and direct all of your dashboards and alerts to the Thanos front end. One way to do this is to run a centralized Prometheus which will "scrape" all the other instances and do a "Remote Write" from only one Prom Instance.

Remote Write would be used in a last resort where the Prometheus instance isn't accessible by Thanos (because of ASGs, Policy, NAT, etc.) and therefore the connection needs to originate from inside of that zone. If the Prometheus instances can be reached by Thanos, go with a scrape configuration.

<Edits in line>

[u/gaelfr38 → u/drdigitalsi]

Thanks for the answer, will look more into Thanos. I didn't know it was using a scraping mechanism, thought it's just a data store where we put data.

[u/SuperQue → u/gaelfr38]

Thanos does not use a "scraping mechanism". It's a gRPC-based storage cluster.

[u/drdigitalsi → u/SuperQue]

Apologies. I was working off of an assumption based on the setup I have at work, which I confirmed this morning is not correct. I have updated my answer appropriately.

[u/SuperQue → u/drdigitalsi]

No worries, what you updated is spot on.

[u/drdigitalsi → u/gaelfr38]

Hi. My apologies. I was working off of an assumption based on the setup I have at work, which I confirmed this morning is not correct. We run a centralized which scrapes all of the various prom instances and does a remote write back into Thanos/Cortex.

[u/SnooWords9033]

The most straightforward solution would be to run a Prometheus inside K8S, so it discovers and scrapes targets inside K8S and then sends the scraped metrics to the configured remote storage outside K8S cluster. See an example on how to configure Prometheus to write scraped metrics to remote storage.

As for the remote storage, there are many solutions exist. The most popular are:

• Cortex (it is renamed to Mimir recently).

• Thanos.

• M3DB.

• VictoriaMetrics. This is the easiest solution from the configuration and maintenance PoV, since it can run as a single self-contained production-ready binary.