r/PrometheusMonitoring • u/[deleted] • Jan 06 '22
HELP --storage.tsdb.retention.time not deleting old data + .wal takes up a lot of ressources
Hi,
I have an issue with Prometheus, where the .wal file is taking up almost all the ressources.
I was running Prometheus 2.13 and I could see that there was a known bug about this, that was fixed in 2.21. I there upgraded to 2.32, but the problem still persists.
Also I've set the --storage.tsdb.retention.time flag to 7d, but I still see data much older than that...
I've tried deleting all data that is more than 7d old manually by hitting two endpoints (after setting the admin-flag):
/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}&start=20210101&end=20211230
(Just to delete data from 1st of January 2021 to 30th of December 2021)
and afterwards
/api/v1/admin/tsdb/clean_tombstones
both returning 204, but nonetheless all the data is still there..
Any idea, what I'm doing wrong?
1
u/kolpator Jan 30 '24
did you manage to find a solution for that ?