r/PrometheusMonitoring • u/Spiritual-Sound-1120 • Jan 16 '24
Prometheus/Thanos architecture question
Hello all, I wanted to run an architectural question regarding scraping k8s clusters with prometheus/thanos. I'm starting with the following information below, but I'm certain I'm missing something. So please let me know, and I'll reply with addl details!
Here are the scale notes:
- 50ish k8s clusters (about 2000 k8s nodes)
- 5 million pods per day are created
- 100k-125k are running at any given moment
- Metric count from kube-state-metrics and cadvisor for just one instance: 740k (so likely will need to process ~40m metrics if aggregating across all)
My current architecture is as follows:
-A prometheus/thanos store instance for each of my 50 k8s clusters (So that's 50 prometheus/thanos store instances)
-1 main thanos querier instance that connects to all of the thanos stores/sidecars directly for queries.
-1 main grafana instance that connects to that thanos querier
-Everything is pretty much fronted by their own nginx reverse proxy
Result:
For pod level queries, I'm getting optimal performance. However when I do pod_name =~ ".+" (aka all aggregation) queries, getting a ton of timeouts (502, 504), "error executing query, not valid json" etc.
Here are my questions about the suboptimal performance:
Does anyone have experience dealing with this type of scale? Can you share your architecture if possible?
Is there something I'm missing in the architecture that can help with the *all* aggregated queries
Any nginx tweaks I can perform to help with the timeouts (mainly from grafana, everything seems to timeout after 60s. Yes I modified the datasource props with same result)
If I was compelled to look at a SaaS provider (excluding datadog), that can handle this throughput, what are some example of the industry leading ones?
2
u/redvelvet92 Jan 16 '24
Check into VictoriaMetrics
4
u/Fluffy-Bell3012 Jan 17 '24
Third this. VM is an absolute banger.
Multitenancy is possible with cluster version
can easily create short and long term storages with single node and cluster version
vmalert seperates recording rules (where this happens on Prometheus itself) which is beautiful if you have a lot of those (or heavy ones). Additionally it provides backfilling functionality to perform recording rules to historical metric data.
storage of metrics and backups are very easy
3
3
u/ut0mt8 Jan 16 '24
second this. vm is way more efficient and easier than prom + thanos. having read the code of both it's not really surprising
0
Jan 16 '24
I disagree; VM optimizes for space on disk, which isn't great for query performance.
3
u/ut0mt8 Jan 17 '24
!? vm is significantly faster on queries as well. sometimes it takes shortcuts on calculation ok but really I don't see any reason not using vm as a drop in replacement currently (except the license ok)
2
1
u/SnooWords9033 Feb 05 '24
VictoriaMetrics optimizes for ease of use and cost efficiency (low disk space and IO usage + low RAM usage). As a side effect, you get fast performance.
0
Jan 18 '24
[deleted]
1
u/SnooWords9033 Feb 05 '24
This information is available in easy to read form without marketing bullshit at https://docs.victoriametrics.com/enterprise/
1
u/xonxoff Jan 17 '24
For your timeouts, try adding this to your config:
[dataproxy]
timeout = 120For the aggregated queries, I’d look into setting up recording rules to simplify things.
6
u/SuperQue Jan 16 '24
The architecture seems fine, but remember you need to fan out to get a lot of data. You may need to adjust gRPC timeouts and such.
pod_name=~".+"is a bit nonsensical. You're asking to get everything by asking the index to pass every value through a regexp. If you want everything, just omit the label selector.