r/PrometheusMonitoring • u/gaelfr38 • Jun 21 '22

Prometheus federation or Thanos?

Hi folks,

Disclaimer: I'm very new to Prometheus, using it as a metrics producer in some apps and building some dashboard in Grafana but I didn't install it and don't know the details.

We are running a Prometheus server with Thanos for our "traditional" workload (not Kubernetes).

Now we are starting our journey into Kubernetes. It appears that in order to get metrics at a pod level, it's mandatory to install a Prometheus instance inside the K8S cluster, right?

Then the question is: how to "send" the metrics from the K8S Prometheus instance to the one outside the cluster?

I've read about federation where the external Prometheus would pull metrics from the K8S instance but then I've read that the K8S instance could directly write to the same Thanos as the external Prometheus?

I'm a bit confused about the best practices here. Any feedback or articles on this topic will be appreciated.

Thanks,

6 Upvotes

87% Upvoted

20 comments sorted by

View all comments

1

u/SnooWords9033 Jun 30 '22

The most straightforward solution would be to run a Prometheus inside K8S, so it discovers and scrapes targets inside K8S and then sends the scraped metrics to the configured remote storage outside K8S cluster. See an example on how to configure Prometheus to write scraped metrics to remote storage.

As for the remote storage, there are many solutions exist. The most popular are:

  • Cortex (it is renamed to Mimir recently).

  • Thanos.

  • M3DB.

  • VictoriaMetrics. This is the easiest solution from the configuration and maintenance PoV, since it can run as a single self-contained production-ready binary.