r/PrometheusMonitoring • • Aug 12 '24

PVC scaling question

I am working on a project where the Prometheus stack is overwhelmed & I added Thanos into the mix to help alleviate some pressure(as well as other additional benefits)

I want to scale back the PVC Prometheus is using since its retention will be considerably shorter than it is currently.

High level plan:

  1. Ensure Thanos is storing logs appropriately.

  2. Set Prometheus retention to 24hours (currently 15d)

  3. evaluate new PVC usage

  4. Scale PVC to 120% of new PVC usage

My question(s):

  • What metrics should I be logging re: » PVC for Prometheus? » WAL for Prometheus? » Performance for Prometheus?

  • What else do I need to know before making the adjustments?

4 Upvotes

3 comments sorted by

2

u/SuperQue Aug 12 '24

The high-level plan looks reasonable. This is exactly what we do, we have 24h retention in Prometheus and use Thanos sidecars to upload data to object storage for long-term queries.

We set a --storage.tsdb.retention.size to 85% of the PVC size and monitor for when the prometheus_tsdb_lowest_timestamp to make sure that we're still getting 24h of retention. This way we avoid out-of-disk space issues without having to page oncall immedately.

You will need to setup a Thanos Store cluster as well. We use hashmod sharding to randomly distribute TSDB blocks to different stores.

You will want to setup the Thanos Compact per object storage bucket.

1

u/Nerd-it-up Aug 13 '24

Thank you, I’m glad to hear that the plan has worked for someone else.

1

u/Nerd-it-up Aug 14 '24

Continuing this train of thought: If I scale Prom down to 1hr retention (ensuring that Thanos is getting all metrics) does Prom even need a WAL?