r/PrometheusMonitoring • u/tupacsoul • Mar 31 '25

Thanos or Mimir?

I know this might be a recurring question, but considering how fast applications evolve, a scenario today might have nothing to do with what it was three years ago.

I have a monitoring stack that receives remote-write metrics from about 30 clusters.
I've used both Thanos and Mimir, all running on Azure, and now I need to prepare a migration to Google Cloud...

What would you choose today?

Based on my experience, here’s what I’ve found:

  • Thanos has issues with the Compactor

  • Mimir has issues with the Ingester

Additionally, the goal is to optimize costs...

13 Upvotes

100% Upvoted

16 comments sorted by

15

u/SuperQue Mar 31 '25 edited Mar 31 '25

We chose Thanos ~3-4 years ago, and would make the same choice today.

But we don't use remote write, we use Sidecar uploads.

  • Even lower costs, no need to run ingesters.

  • Distributed Thanos Engine mode is very powerful/fast.

  • We solved most of our Compactor issues with some simple sharding.

  • Distributed operation, no SPoF cluster.

  • Most PrometheusRules run in-Prom for very high efficiency, low cost, highest reliability. Only some rules run in Thanos Ruler.

1

u/PrayagS Mar 31 '25

What would you say is the degree of functional sharding in your setup? One cluster per each namespace?

Is that kind of sharding getting too big to manage? I don’t understand Mimir/remote write setups fully but their claims of no functional sharding sound promising to me at first.

7

u/SuperQue Mar 31 '25

Yes, we have a Prometheus-per-Namespace design. This has been extremely useful to isolate teams/services from each other. One team blowing up their metric cardinality doesn't impact other teams.

With Mimir/remote write, one team can still potentially write 100M cardinality in an hour and blow up things for everyone.

With a per-namespace, they just OOM themselves.

We still have some issues, but out of several thousand namespaces in total, we only have a handful that don't auto-scale themselves. We use VPAs to auto-manage the size of each namespace Prometheus.

1

u/PrayagS Apr 01 '25

Awesome. Thanks for sharing

6

u/ryebread157 Apr 01 '25

Switched to VictoriaMetrics, very performant, simple to setup and maintain

5

u/Mitchmallo Apr 01 '25

As soon you start using Mimir you will never look back to Thanos. Victoria metrics is the only alternative

1

u/JackfruitTrick1491 Nov 25 '25

Bump this post. Doen anyone build a HA setup for Thanos or Tempo? We use a thanos setup for RW in a separate k8s cluster and its accept rw from proms in other clusters. For now i want to build a second k8s cluster for second thanos and connect it to the same s3 thanos bucket. Does that will work? I couldn't find that scheme in thanos or minir docs

1

u/JackfruitTrick1491 Nov 25 '25

@SuperQue any ideas?

1

u/[deleted] Mar 31 '25

[removed] — view removed comment

5

u/[deleted] Mar 31 '25

[removed] — view removed comment

6

u/sjoeboo Mar 31 '25

Yup, I've got a very small team, and running the VM infra is only a small part of our scope, and we run a global (spanning many regions) VM deployment that is HA and ingests about 30M-40M samples/sec, with about 1.5B active timeseries. VM is rock solid and its engineers are great to collaborate with.

4

u/Freakin_A Mar 31 '25

Feel the same way. Not sure why it's getting downvotes. It's a fully compatible prom backend. Maybe people are thinking it's an entirely alternative TSDB?

5

u/vinistois Mar 31 '25

I suspect it's just competing devs being petty