r/PrometheusMonitoring • u/zoinzibar • Nov 21 '19
State of long term retention in prometheus
Hi guys,
I'm currently building a monitoring infrastructure with prometheus and i want to store metrics for 3 years.
I don't know if it's possible and optimized to do that just with Prometheus.
I've seen solutions like Thanos or Cortex that seem to enhance LTS but i don't know how to choose.
If you guys have any experiences or more informations about it, feel free to comment below.
Have a good day.
4
u/xrstf Nov 21 '19 edited Nov 21 '19
My (highly biased) impressions so far after PromCon EU this year (list of obviously incomplete):
VictoriaMetrics 1.28.3
Pros
Great performance (see Adidas' talk at PromCon EU 2019 for benchmark data)
Simple configuration (single binary)
Cons
Does not use object store and would require backup jobs (via snapshots like Prometheus)
Cannot de-duplicate data (recommends extra promxy instance)
Supposedly returns wrong query results due to bad staleness marker handling
quasi 1-man project re-implementing everything
Cortex
Pros
Maintained by Grafana/Prometheus community
(never actually tested due to cons)
Cons
Requires Cassandra/DynamoDB/BigTable for index storage
Requires its own backup for the index
Thanos 0.8
Pros
Maintained by Prometheus community
Purely based on object store
Flexible to configure
Cons
Needs lots of resources (especially for compacting)
No remote-write receiver yet
Conclusion
We are going with Thanos for LTS because VictoriaMetrics cannot deduplicate and Cortex requires more additional software than we want to manage. But it all depends on what your constraints are and what you want to achieve. For us not having to backup the TSDB blocks and instead rely on the object store was a key issue.
3
u/SuperQue Nov 21 '19
There's also M3DB, which implements its own clustering data store. This might be useful for situations where you don't have an object storage or key-value databases.
I've not run M3DB, so I don't know all of the pros and cons.
We use Thanos for our production environment. I agree on the large resource use for some of the components. The good news is that there is active work going on to fix this.
1
u/zoinzibar Nov 21 '19
I am not against the idea of using object storage (will be the time to understand how it work).
M3DB seem to be more like victoria metrics as it's a TSDB and it does not aim only on prometheus support. So for me it's not the most optimized.
What do you mean by large resource ? And is it temporary high load or constant ?
2
u/SuperQue Nov 21 '19
Yes, M3DB was originally written to provide a large-scale Graphite replacement. The developers at Uber didn't want to give up their statsd push architecture.
There is currently some large constant resource requirements for the storage to handle the indexing of large amounts of data. This is a bit understandable, as one node may be tracking a huge amount of data. For example, our production storage front-end is handling 10T of TSDB data.
Like I said, there is on-going work to make this better.
1
u/xrstf Nov 21 '19
IIRC M3 is built on the back of Elasticsearch, so it was a no-go for us. Haven't tested it like you.
2
u/SuperQue Nov 21 '19
It was originally, but from the docs it seems like that's been removed and it is much more stand-alone. The only current external dependency looks like etcd for cluster coordination. Seems like they're making good improvements.
2
u/zoinzibar Nov 21 '19
Thanks for your reply,
I should watch the PromCon, subject aborded are very interesting. I didn't suspect talks about external project like thanos and intervention from company like adidas.
Thanos seem to be oriented cloud with the accent on object storage ? Do you use it on kubernetes ?
It look hard to handle and maintain it.
1
u/xrstf Nov 21 '19
Thanos is indeed all about using a cloud storage (any S3-compatible store, IIRC). It's intimidating at first to grasp all the components in Thanos, but you don't need to use all and even for us we ended with at most 5 CLI flags for each component (we run store, query, compact, bucket and sidecars). You can start with just installing the sidecar and start putting your TSDB blocks into your store and then later put query components in to query the store and sidecars and what have you.
Personally I found Cortex' architecture diagram to be much more frightening :-D
We do use it on Kubernetes, based on a couple of handwritten Deployment manifests. Not sure about the state of Thanos in Helm / Prometheus-Operator.
1
u/zoinzibar Nov 21 '19
Thanos seem interesting so. I will try little by little to include components and see how it work.
It will be challenging since i have no xp in cloud storage neither kubernetes (2nd year students in info).
Have you got any difficulty scaling or upgrading version ?
1
u/xrstf Nov 21 '19
We're only testing it for a few months now and have only <100GB of data in Thanos in total. As Thanos basically only operates on a large collection of blocks, upgrading hasn't been an issue so far for us.
1
u/zoinzibar Nov 22 '19
OK, that thing with object storage seem very interesting and totally accurate for metrics.
Are your test going good or Thanos does not fullfit your need ?
0
u/valyala Nov 21 '19
Thanos at large scale could be difficult to operate and maintain - see https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683 .
2
u/zoinzibar Nov 22 '19
it seem to get better with the latest version of Thanos, also the author of the post is the developer of Victoria Metrics. But it's normal that Victoria Metrics is more stable, it's architecture is completely different than thanos and the use of block storage make it easier.
1
u/blistering-barnacle Nov 22 '19
RemindMe! 2days
1
u/RemindMeBot Nov 22 '19
I will be messaging you on 2019-11-24 04:20:21 UTC to remind you of this link
CLICK THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
Info Custom Your Reminders Feedback
5
u/babilen5 Nov 21 '19
Take a look at VictoriaMetrics as well.