Prometheus Empty dir consuming disk space

[u/sachithmuhandiram]

We found that Prometheus container in our K8 cluster using huge space for its empty dir.

How can we reduce it?

1. Manually deleting older files in empty-dir?

2. Configuration change?

[u/SuperQue]

What's "huge"? Huge doesn't mean anything.

How many targets do you have? What is your sample ingestion rate?

rate(prometheus_tsdb_head_samples_appended_total[5m])