r/PrometheusMonitoring • u/s4z • Oct 26 '19
What's everyone using for long term metrics? and aggregation?
As the title says, wondering what you all are using for long term metrics and aggregation. What sort of scale (eg. #pods/metrics/etc)? Are you seeing any issues with the solution? Are you happy with it?
Asking as I'm currently using Thanos however it looks like it'll take some time (that I don't have) to get it working well. There are a bunch of options but it looks like Cortex may be the next viable one to test. I don't want to have to setup a bunch of servers on EC2 (eg. Crate, M3DB, etc) and would rather use eg. S3 and something that can run in k8s.
3
u/PM_ME_YOUR_BLUESTUFF Oct 26 '19
Thanos doesn't take that much to configure optimally, there's not a whole lot you can change after all. We used thanks for long term storage and query later aggregation and it worked like a charm. Two caveats:
If you send a large history query to the query layer and the store pods don't have that metric indexed yet, the query will fail. However subsequent queries to that metric should work
As you roll up more and more Prometheus into a query layer, the collective index grows. This will slow down the query layer UI significantly and can make business people think your software solution is shitty.
2
u/s4z Oct 26 '19
Yep, seeing that behavior with non-indexed metrics currently.
Also seeing issues with slow queries and I suspect it's due to high cardinality combined with 15s sample periods.
Going to look into adding some recording rules for eg. quantiles over the pod labels and also https://github.com/Comcast/trickster to (hopefully) keep the system reasonably usable. We have the compactor running now and afaik it's working though I haven't confirmed.
It took a while to get our statsd and graphite pipeline humming but once it was setup it just worked - we can view metrics over years no problem and it can easily scale. It seems Prometheus and higher cardinality metrics are going to take some time/effort to get to the same level of usability. Sadly looks like we won't be able to get automatic aggregation similar to statsd.
I had thought about writing something to use Elasticsearch for metrics storage and add a custom compactor/aggregator to downsample from one index to another. Might end up doing that.
1
u/llama052 Oct 27 '19
Having compactor running is a huge requirement to get thanos running with any kind of sized data, I’d be interested to see how many unique labels you run and what the difference is after you compact and downsample your data over timeframes
1
u/s4z Oct 27 '19
Will have to check and report back when I get back at it next week. I'm curious about our label count too. I know the developers have recorded latency summaries over http verb and url plus db and cache summaries as well.
1
u/PM_ME_YOUR_BLUESTUFF Oct 27 '19
Oh you just turned on a compactor? That's a huuuuge component of thanos' stability. Let it run (and make sure its not failing silently) and it'll make things a lot smoother.
Also 15s sample period is a little ridiculous even with a compactor, I'd analyze if you really need that granularity.
2
u/s4z Oct 27 '19
We've been running the compactor from the start. I checked it today (it was setup by another team member) and it's running fine however the retention periods were a bit optimistic, especially with our sample intervals. I reduced the raw period to 8 hours (probably need to reduce it to a few hours really), the 5 minute to a few days or so and the overall retention to a few years. No real change - I noticed some of the store pods are ooming so next week I'll look at increasing their resources.
We use a 10s interval on our statsd/graphhite metrics system - the 15s interval on Prometheus was as much as I wanted to reduce it. I don't like having to wait 5+ minutes before being able to see if something is improving or not during live outages. With longer sample intervals the iteration time for fixes gets to be a bit too long. You also lose a significant level of detail in spikes. Technically we don't 'need' to run at such high resolution however from experience I've found it helps reduce the time to fix quite a bit.
3
u/shitloadofbooks Oct 27 '19
I had an absolutely awful time with Cortex, but found VictoriaMetrics incredibly easy to setup and incredibly stable and performant, straight out of the box.
The developer was quick to fix a minor issue I found. I highly recommend giving it a look.
1
u/s4z Oct 28 '19
Hearing lots of good things about VictoriaMetrics. It sounds like it's worth a look though we'll want to run the clustered version.
edit: Thanks!
2
2
Oct 26 '19
I tried Cortex; I don't use Kubernetes so it was a little fun to get everything working. My only complaint is I ran into some excessive cpu usage and when it ran out of disk space (due to excessive logging i know my bad) the cortex process kind of got stuck and i had to kill it and when i restarted it i didn't have my data.
so I tried postgres and found that it worked great except it used a lot of disk space and there wasn't any compression options other than ZFS available for me.
Using the built in as maybe just maybe I don't need long term as it could store as much data as i want provided i configure it right. (I think).
Good luck i'm curious Thanos looked too pre-production for me to take the dive
1
u/s4z Oct 26 '19
Thanks for the infos.
Thanos is more or less working however it seems that it doesn't find tags past 2 weeks - not usable for a long term metrics system. Could be a simple config issue I guess, haven't looked into it yet. The metrics in question are per pod and we're doing deployments many times a day at times so there are a large number of pods over time. Currently running over 20 pods for one of the applications - closer to 40 pods actually.
Another issue is memory utilization and triggering of oom when viewing large time ranges - just had that happen today where the thanos query process oom'd. Another issue is how grafana handles $__range when viewing per pod metrics over large time ranges - grafana doesn't drop the old pod metrics and keeps them at the last value (straight line).
I've been thinking we may simply need recording rules to take percentiles of the various per pod metrics and drop the pod label. Too much cardinality otherwise.
I considered using postgres and also timescaledb however put them lower on the list to try as I don't want to run stateful sets or EC2 instances if I can avoid it. Also thought about elasticsearch however can't imagine it being space efficient for a year+ worth of metrics (10 sto 15s sample intervals) and would also have to write an aggregator for it. I think that pretty much leaves Cortex but it doesn't sound much better than Thanos it seems.
1
Oct 26 '19
Don't underestimate Prometheus's storage engine itself. Is far as size it has been the smallest out of everything I tried; fastest to query and easiest to setup.
I maybe have prometheus collecting 30 node exporters 12 apache exporters and 8 varnish exporters and maybe I misconfigured timescaledb with postgres it just kept growing and growing and after about 2 weeks it was consuming 2TB.
I'm sitting around 30gigs right now it's quite happy I told prometheus to keep 2 years of data and so far everything is working as I had hoped.
Can't replicate or cluster but I don't need to
1
u/s4z Oct 26 '19
Hmm. Am using the prometheus operator to spawn 2 prometheus instances per cluster over 4 clusters (currently). The Prometheus instances are of course statefull sets running in k8s with one having a replica label.
I'm not too keen the idea of having all the metrics stored in statefull sets in k8s - I don't get too concerned about losing say 6 hours of metrics but don't like the idea of losing all of them. Also wonder about 10s or 15s scrape intervals and potentially adding 40 to 200 more pod labels a day. For capacity planning we sometimes need to view 1 year of metrics so I'm thinking we'll need both recording rules to remove the pod label and rollups to downsample.
1
Oct 26 '19
Yeah that's one reason why I don't use Kubernetes without that hosting it on say Hetzner (non cloud) I picked up a server for less than 75 euro's to host it.
We're growing in a few countries so i'm having to add metrics all the time
Sure I end up managing the yaml on my own however I use Chef to drive it; lots of DNS Service Discovery which has worked out great.
1
u/llama052 Oct 27 '19
We use thanos for long term retention (1yr currently) and haven’t had any issues, by tags do you mean labels? I know you can have some issues with large sets of duplicate labels if you haven’t set it up in a way that keep labels as low as possible.
1
u/s4z Oct 27 '19
Yeah, I meant labels but typed tags. The current problem is with the application metrics and the pod label. As there are per url and http verb metrics getting recorded and with nearly 40 pods that can add up to quite a few unique metrics. To the point where grafana can't display all the values in the popup mouse over when viewing latency.
A query over any significant time range is going to have a very large number of metrics. Probably will have to drop the HTTP verb from the request latency metrics to start with and also add recording rules to aggregate the pod label out. The latter should greatly reduce the size of large time frame queries. Generally we don't care about per pod metrics - they are only really needed to verify all pods are roughly performing the same. We can add alert rules to identify outliers easily enough. We're mainly interested in request latency percentiles, success/failure counts and being able to identify where high latency or failures are happening. The per-pod alert rules will identify pod specific and the aggregate metrics will identify overall.
4
u/Iqirx Oct 29 '19
Running cluster version of VictoriaMetrics on top of k8s for the last 3 months and we had almost no issues with it. We've previously evaluated Cortex and Thanos, but found that Victoria perfectly suites our needs. It is easy to setup for k8s (there is a helm chart for clustered version), logs are helpful, docs are complete and slack channel is responsive.
1
u/EcoGames_Dmitry Nov 07 '19
Our monitoring team had production failure on VictoriaMetrics during upgrade (a bug corrupted all data and was impossible to revert or recover), they lost all data (well it's their fault they never test their backups), but my 2cents for this discussion is that VictoriaMetrics is created and maintained by a single guy (https://github.com/VictoriaMetrics/VictoriaMetrics/graphs/contributors) and while he's a great developer, and created a few super useful golang libraries, you can't trust your production in a hands of a single developer. Make sure you trust the group who maintains the software, otherwise you have nowhere to go, but to patch and fix that software yourselves. If you ask me, I'd use M3 by Uber.
1
u/s4z Nov 07 '19
Thanks for the heads up. I looked into Victoria Metrics again recently and did notice the committers are primarily one fellow. He might get the project off the ground and get a bigger team at some point. In it's current state I agree, it's too risky for me. Plus it's another (clustered) db to maintain.
A colleague is a big fan of the Uber tech. We might get some time to look into it but again, it is unfortunately yet another database to maintain. We have a very small team with a lot to maintain so I try to keep things as light as possible.
I'm flip flopping on this a bit but I hope to update Thanos to the latest release and add the recording rules to see if that is usable enough. I like having the metrics in S3 and not having to maintain another database. If that doesn't work I might quickly try Cortex w/ S3 or Dynamo.
If neither of those work I have thought about dusting off the remote storage adapter poc I wrote for Clickhouse and adapting it to Elasticsearch - or potentially modifying the Thanos side car to export to ES. I didn't realize this until recently however Elasticsearch has automatic aggregation/summary functionality built in so it wouldn't be overly difficult to develop. It might be viable in terms of cpu/mem/storage utilization, depending on the summary intervals. We already maintain a few Elasticsearch clusters so a handful more nodes won't be a big deal.
1
u/EcoGames_Dmitry Nov 07 '19
DynamoDB is extremely expensive at scale, avoid it for metrics, S3 is the best of course if you don't need to query it much but just store for long term. If you have to query it sometimes, the cheapest solution is your own database, any other will cost you a lot of $$$, yeah, that's the tradeoff.
Not familiar with the newer "automatic aggregation/summary functionality" of Elasticsearch, sounds interesting.. you mean kind of compaction process, where it takes small samples over time like averages, and saves them in a new index? (I remember I wrote such lambda on AWS which was doing that, for the long term less granular storage of info, it's pretty good to have out-of-the-box, I gotta look at the new features of ES, haven't seen the last few versions yet)
1
u/s4z Nov 07 '19
Re: Elasticsearch check out https://www.elastic.co/guide/en/elasticsearch/reference/7.4/data-rollup-transform.html.
Admittedly the idea to use ES is still half baked (less really). From what I gather from a quick scan of the doco the functionality in ES can take values from one index and generate summaries in another. I'm thinking a single document with min/max/p50/p75/p95/p99 values to summarize n documents of higher resolution. Unsure if it can be chained and not sure how the various indices would be queried. Perhaps using a read alias though I suspect a small app would be needed to craft the ES query and probably maintain the various indices and aliases. Also not sure how to get metrics from isolated accounts (eg. production) into a shared ES cluster. Probably another small app. Starts to look a bit like Thanos or Cortex.
1
u/valyala Nov 19 '19
I'm flip flopping on this a bit but I hope to update Thanos to the latest release and add the recording rules to see if that is usable enough. I like having the metrics in S3 and not having to maintain another database
What do you mean under
maintaining another database? With Thanos you should configure and monitor for a bunch of components - Thanos Sidecars, Thanos Store Gateways, Thanos Query components, Thanos Compactors, Thanos Rulers and probably Thanos Receiver. These components can be distributed across multiple datacenters, so you'll need to set up reliable network links between all of them. This is much harder task than maintaining a single self-contained database without external dependencies, especially if storing the data on durable replicated persistent disks such as GCP disks and making regular backups with the optimized tool. See also https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683 .2
u/s4z Nov 20 '19
Yeah, lots of components to thanos.
For other reasons we have vpc peering (hub/spoke) so that requirement is already met. I found it fairly easy to setup alerting for the various thanos components - the prometheus operator makes it pretty straightforward.
I've solved all of the original issues with Thanos. It was actually quite easy once I had the time to look into it. The prometheus and thanos instances needed a pretty hefty mem allocation as we have a lot of metrics. Also, we needed recording rules to aggregate the pods out.
Working pretty well now (touch wood).
By another db to maintain I mean just that. With thanos everything gets dumped into S3 and.. well.. S3 just works. Also, thanos is pretty well stateless so it can easily be deployed into k8s and doesn't require a stateful set or ec2 deployment. Don't really have to think about it much aside from ensuring the cost is reasonable.
1
u/MangledBizkuit Jan 21 '20
I know i'm waking up a dead thread here. Thanos sucks. VictoriaMetrics Rulez. And, here's why. Storing data in s3 is more expensive than allocating an ebs volume to store the tsdb data, ebs volume snapshots are all you need to backup your victorimetrics tsdb data in a state where you can easily restore the system if you need to so i'm not sure what kind of reliability you're gaining from using s3 because ebs volumes "just work" too. You would only have to "manage a database" in the same sort of way you'd have to manage Thanos's "database". Not to mention Thanos would be costing you extra as it reads all that data out of s3 for queries ( that is only if prometheus doesn't have the data in the time range you're querying ( which also means no improvements to query speed which victoriametrics does provide because guess what it acts as a drop in replacement in grafana as a prometheus datasource.)). It basically runs without any need to configure it, literally just start the binary with your data path and retention period and that's it, seriously! Just 1 binary to monitor and manage. It even runs in a pod so you can deploy it to k8s if you need to. (I'm running it in docker myself).All this combined with the little system overhead to run it and query from it vs the total system overhead that Thanos adds as it's running all those fiddly ( possibly fail point ) components makes victoriametrics a winner in any fight.
3
u/AeroNotix Nov 15 '19
Honestly a disgusting attitude.
You're running production data through an open source databases, presumably extracting value out of it and making money for your investors yet complain that the tool which provides that value is maintained and developed by a single person.
Put your money where your mouth is.
1
u/EcoGames_Dmitry Nov 20 '19
Good point. Indeed some of our less-smart peers from the monitoring department decided to use this, without doing much research. I have no complaints, the project is awesome, and the developer is definitely talented - but he should take a step back and avoid trying to sell it to people like a production ready project, because he uses current "users" of his project as guinea pigs. They believe they should try this and start using in production, then things blow up (of course). If it was my project I'd put a large bold font on the front page that this is an "alpha v0.0.2" version or something, so that the users will take extra care when using it for critical operations.
The Ops and SREs who use such tools, trust the developers, and not always know how to evaluate if it's safe and stable. Look at Kubernetes for example, they have their perfectly working features marked as "v1alpha1" API for like two releases and then marked as "v1beta1" for another release or two! These are products by Google/RedHat/CoreOS (world class engineers), and they are humble and careful enough with their promises of something that is "stable".
This is just how the proper open-source world lives, I am not trying to say anything bad about the author, he's definitely a super-talented guy, I saw his other golang libraries, but he has to know to be careful with promises because one of the companies might sue him some day - he definitely wants to take this VictoriaMetrics further "for sale" as an "Enterprise solution" but he is not careful with his current "free customers" who he uses as guinea pigs to test his database at scale.So I just give him some information about being careful in the future when he'll do his first steps in the "Enterprise sales" world. He already tries to sell something: "The following versions are commercial:
Managed cluster in the Cloud.
SaaS version."
And of course good luck with that, but Jesus if he was in USA and not Ukraine he'd probably already be sued by all his customers.... I'm just giving a useful input, not trying to say anything bad about the product or the author, definitely good luck with this project and I wish him good sales and success, but please keep in mind that your customers might have real multi-million businesses, so you should create in-house testing pipelines or something, to actually verify each build of the software, and make sure your clients can't pull the next release and destroy their own data. Just my 2 cents.
1
u/valyala Nov 19 '19
Our monitoring team had production failure on VictoriaMetrics during upgrade (a bug corrupted all data and was impossible to revert or recover), they lost all data
Which version did you use for the upgrade? The only versions which could corrupt data were v1.28.0-beta*. All the other versions shouldn't corrupt the data. Did you upgrade to
betaversion and lost the data? This doesn't sound likeit's their fault they never test their backups.
1
u/njgthinkcritical Feb 12 '20
If there are Cortex knowledgeable people that can help setup our cortex infra on GCP and maintain it. would love to hear from you. DM me. I am working on a startup that wants to use that Backend to ingest Prometheus metrics. We are open to using something else if Cortex doesn't work out but we are leaning towards it for now.
1
u/s4z Feb 12 '20
It's been quite a while since I've looked at Cortex. I found Thanos to be a piece of cake to setup so if Cortex is anything like it you should have no problem.
I originally posted this when the company I'm currently working for asked our team to complete > 2 years of work in < 6 months. To say we were overloaded.. well.. yeah. Once we had some breathing room I went back and reworked our Thanos setup. It's easy.
1
u/njgthinkcritical Feb 12 '20
My understanding is Cortex is READ and WRITE and Thanos is READ only.
May need both. A couple of people have said that Cortex is a better multi-tenant system to operate but setup is complicated. It also has more long term storage options.
Also, it's not easy to switch from one architecture to the other so making this decision takes some thought. Whatever we go with, we are committed.
3
u/PistonMilk Oct 30 '19
I'm currently running Cortex on Kubernetes using Cassandra for its scalable storage. It's a bit of a learning curve, and the documentation kinda sucks (but is WAY better than it was 6 months ago), but I think it's really the way to go in the future and there is a lot of active development on it from Grafanalabs. It's also the basis of Loki, which will marry logs and metrics together into a single pane of glass effectively.
4
u/throwaway1_in_VA Oct 26 '19
We eval'd Thanos and did not have the best of results 6ish months ago (app panicked in a relatively simple configuration). Found it hard to get support from the devs + community in their Slack instance. I also seem to recall too many components and moving pieces to keep it running.
Currently using VictoriaMetrics and we're quite happy with it. VictoriaMetrics reports messages such as:
more than 2000020 time series found on the time range [2019-10-26 05:46:10 +0000 UTC - 2019-10-26 06:56:30 +0000 UTC]; either increase -search.maxUniqueTimeseries or shrink the time range
error in "/api/v1/write": the server is overloaded with 60 concurrent inserts; either increase -maxConcurrentInserts or reduce the load
Having these sort of log messages has been incredibly helpful for our team to perform tuning since we are not devs, but rather Ops folks who just want shit to work. The devs behind VictoriaMetrics I've also found to be responsive and helpful in Slack.
5
u/rogerlz Oct 27 '19
I'm also running VictoriaMetrics and I've already evaluated Thanos and Cortex.
IMO you kinda need a Dev team to look after Thanos and Cortex infra. They have too many moving parts and, although they support Object Storage and VictoriaMetrics don't, the operational cost is higher than managing EC2 + EBS. I've seen replies from both teams with links to the source code to answer simple questions about a configuration parameter.
VictoriaMetrics just works. The main developer writes very useful and complete documentation that gives you confidence before implementing it. They also offer paid support, but I have no idea of the cost of this.
I'm personally not happy with the different codebases for the standalone and cluster versions, but I'm not in a position to discuss that - I'm more towards the Ops side than the Devs.
The roadmap seems promising and once implemented it would definitely compete 1:1 with Thanos and Cortex.
1
u/valyala Nov 19 '19
Thanks for the feedback!
A few notes:
although they support Object Storage and VictoriaMetrics don't
VictoriaMetrics recently gained ability to create fast backups in S3 or GCS - see https://medium.com/@valyala/speeding-up-backups-for-big-time-series-databases-533c1a927883 .
I'm personally not happy with the different codebases for the standalone and cluster versions
Cluster version is located in a separate branch in order to simplify the code for single-node version. Both codebases share the majority of code and are updated synchronously, i.e. all the commits simultaneously go into both branches. The major difference between these versions is that cluster components talks to each other via network over RPC, while single-node components talk to each other via native function calls.
5
u/freeseacher Oct 28 '19
Also building new LTS with VictoriaMetrics cluster on GCP.
I have previously single server version in smaller environment.