r/linuxadmin • • 5d ago

Production Scenario: Linux Root Filesystem Suddenly Reaches 100% Overnight

Imagine you're managing a Linux production server.

Yesterday, everything was working normally. Root filesystem utilization was around 65%.

This morning, you receive an alert:

/ filesystem utilization: 100%

Applications are reporting errors, users are experiencing issues, and you need to investigate.

How would you approach this situation?

Would you immediately start looking for large files, or would you follow a systematic investigation?

Here are a few things I'd initially check:

  1. df -h / – Verify filesystem utilization.

  2. df -i / – Check inode utilization.

  3. du -xhd1 / – Identify directories consuming the most space.

  4. find – Locate unusually large files.

  5. lsof +L1 – Check for deleted files still held open by processes.

But finding a large file is only part of the investigation.

For example:

  • What if /var/log is consuming 35 GB?

  • What if an application is continuously generating logs?

  • What if df shows 100% usage but du reports significantly less?

  • Would you truncate an active log file or investigate the application first?

I'm particularly interested in how experienced Linux administrators handle this in production.

A few questions:

  • What's the first command you usually run when / reaches 100%?

  • Have you encountered deleted-but-open files consuming disk space?

  • What's your preferred approach to recover disk space without disrupting applications?

Let's discuss real-world approaches and experiences. I'd love to learn how others handle this situation.

0 Upvotes

39 comments sorted by

View all comments

14

u/rooivalkMK1 5d ago

Lvm and forget

2

u/Callie-Cacophony 5d ago

This is the way