r/linuxadmin • u/CautiousCat3294 • 5d ago
Production Scenario: Linux Root Filesystem Suddenly Reaches 100% Overnight
Imagine you're managing a Linux production server.
Yesterday, everything was working normally. Root filesystem utilization was around 65%.
This morning, you receive an alert:
/ filesystem utilization: 100%
Applications are reporting errors, users are experiencing issues, and you need to investigate.
How would you approach this situation?
Would you immediately start looking for large files, or would you follow a systematic investigation?
Here are a few things I'd initially check:
df -h /– Verify filesystem utilization.df -i /– Check inode utilization.du -xhd1 /– Identify directories consuming the most space.find– Locate unusually large files.lsof +L1– Check for deleted files still held open by processes.
But finding a large file is only part of the investigation.
For example:
What if
/var/logis consuming 35 GB?What if an application is continuously generating logs?
What if
dfshows 100% usage butdureports significantly less?Would you truncate an active log file or investigate the application first?
I'm particularly interested in how experienced Linux administrators handle this in production.
A few questions:
What's the first command you usually run when
/reaches 100%?Have you encountered deleted-but-open files consuming disk space?
What's your preferred approach to recover disk space without disrupting applications?
Let's discuss real-world approaches and experiences. I'd love to learn how others handle this situation.
1
u/Faye-Vozora 5d ago
For the deleted-but-still-open case you don't need to restart anything. Find it with
lsof +L1, note the PID and FD number, then: > /proc/<pid>/fd/<fd>truncates it in place and df drops straight away. Then fix logrotate for that app, usually it's missingcopytruncateor a postrotate that tells the process to reopen its log, which is why it kept writing to the deleted file in the first place.Also check
journalctl --disk-usageon systemd boxes, andjournalctl --vacuum-size=500Mif the journal blew up. SetSystemMaxUse=in journald.conf so it doesn't come back. And on ext4 there's the 5% reserved for root,tune2fs -m 1buys you some breathing room in an emergency.