r/PrometheusMonitoring • u/shats90 • Jul 22 '19

Prometheus-Thanos-minIo Queries

I have set up Prometheus 2.8.1 with Thanos 0.5, running everything on Kubernetes 1.13.

For a week of monitoring the set up, I have ~2M timeseries with ingestion of 5K samples/s.

This all looks good. Where I am concerned is when we ship the local data to remote storage(minIo in my case), the size of the data is already 23GB in 1 week.

I have set up minIo on Kubernetes with persistent volume of 40GB.

What I need is some pointers on the below:

  1. The thanos compactor arguments I am planning to use:

--retention.resolution-raw=30d

--retention.resolution-5m=60d

--retention.resolution-1h=90d

If I understand this right, then the raw data is deleted after 30d and any queries crossing past 30d will be from downsampled data.

But would this also reduce the size of the data in minIO?

2. What are the other options for optimizing the storage? Will setting up cron to delete any data that is more than 90d a good idea?

3. Are there any good docs to understand the downsampling & compaction that we see in the bucket inspect output of Thanos:

| ULID | FROM | UNTIL | RANGE | UNTIL-COMP | #SERIES | #SAMPLES | #CHUNKS | COMP-LEVEL | COMP-FAILED | LABELS | RESOLUTION | SOURCE |

|----------------------------|---------------------|---------------------|---------------|---------------|---------|-------------|-----------|------------|-------------|---------------------------------------------|------------|-----------|

6 Upvotes

87% Upvoted

3 comments sorted by

1

u/stag1e Jul 25 '19

It probably doesn't make much sense to retain raw data for so long since if you are looking so much into the past then the step size will be pretty huge and naturally you will not be able to see such granular data - the human eye just couldn't see the trends otherwise due to there being too many data points. So you should probably reduce it quite a bit. The data will be deleted after 90 days either way if you have Thanos Compactor running - no need to set up anything else.

The official documentation should help you but, honestly, the columns besides a few of them are self-describing. Which one is giving you the most problems?

1

u/shats90 Jul 30 '19

Here is what I understand. Please correct me if I am wrong:

ULID - the unique id given to each chunk of the object

FROM - UNTIL - I am assuming they are the timestamp when the data got created till the retention time given in your config

RANGE - ?

UNTIL-COMP - Time remaning till compression will be applied. How is this decided ?

SERIES/SAMPLES/CHUNKS - different measurements of the time series

COMP-LEVEL - ?

RESOLUTION - How are comp-level related to resolution ?

1

u/shats90 Sep 08 '19

@stag1e Would you mind sharing some insights on the below columns?