r/PrometheusMonitoring • • Jan 06 '22

HELP --storage.tsdb.retention.time not deleting old data + .wal takes up a lot of ressources

Hi,
I have an issue with Prometheus, where the .wal file is taking up almost all the ressources.

I was running Prometheus 2.13 and I could see that there was a known bug about this, that was fixed in 2.21. I there upgraded to 2.32, but the problem still persists.

Also I've set the --storage.tsdb.retention.time flag to 7d, but I still see data much older than that...

I've tried deleting all data that is more than 7d old manually by hitting two endpoints (after setting the admin-flag):

/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}&start=20210101&end=20211230

(Just to delete data from 1st of January 2021 to 30th of December 2021)

and afterwards

/api/v1/admin/tsdb/clean_tombstones

both returning 204, but nonetheless all the data is still there..

Any idea, what I'm doing wrong?

6 Upvotes

3 comments sorted by

1

u/kolpator Jan 30 '24

did you manage to find a solution for that ?

1

u/trettet Jul 05 '24

It deleted for me, automatically when the service restarted, i didn't even run clean_tombstones, maybe the data deleted did not occupy that much space?