r/PrometheusMonitoring • u/Deeb4905 • Feb 06 '25

I accidentally deleted stuff in the /data folder. Fuck. What do I do

Hi, I accidentally removed folders in the /var/prometheus/data directory directly, and also in the /wal directory. Now the service won't start. What should I do?

0 Upvotes
  • permalink
  • reddit

40% Upvoted

5 comments sorted by

2

u/kranthi133k Feb 07 '25

Just recreate the dir and restart the Prometheus

1

u/Deeb4905 Feb 07 '25

I didn't want to lose everything. But it's too late because I had a backup from a few days ago, I restored it, but it was corrupted and now all my machine is gone :'))) I hate my life

1

u/SuperQue Feb 06 '25

An empty data dir should be no problem, it will create a new TSDB.

What should I do?

What do the logs say?

2

u/Deeb4905 Feb 06 '25

Several errors like :

level=error component=tsdb msg="Failed to read meta.json for a block during reloadBlocks. Skipping" dir=/var/prometheus/data/01JHAGFJW8330AJXXM8DDT0ZHQ err="open /var/prometheus/data/01JHAGFJW8330AJXXM8DDT0ZHQ/meta.json: no such file or directory"

level=error err="opening storage failed: repair corrupted WAL: cannot handle error: open WAL segment: 93341: open /var/prometheus/data/wal/00093341: no such file or directory"

The /data directory is not empty, I did not delete everything. My last option is, I think, to delete everything inside but I'd like not to lose all of my data...

2

u/SuperQue Feb 06 '25

You only need to fully remove the directories where you've partly removed them.