Skip to content

$ cat ./posts/storage/finding-what-filled-the-disk.sh

-rw-r--r-- 1.6K #perf #storage

Finding what filled the disk

by chris · 2 min read · /storage


The disk is full, du -sh / adds up to half the capacity, and nothing looks wrong. This is the most common storage puzzle there is, and it has three usual answers.

1. Deleted files still held open

sudo lsof -nP +L1 | head

A link count of zero with a live file descriptor means the space is allocated to a process, not to a path. du cannot see it because it walks directories. Restarting the holder — usually a log writer that was rotated incorrectly — returns the space instantly.

2. Files hidden under a mount point

sudo mkdir /mnt/root-check
sudo mount --bind / /mnt/root-check
sudo du -xh --max-depth=2 /mnt/root-check | sort -h | tail

If something wrote to /var/lib/data before the real volume was mounted over it, those files are unreachable through the normal path and invisible to every tool that walks it. A bind mount of the root exposes them.

3. Inodes, not bytes

df -i

A mail queue or session directory with millions of tiny files exhausts inodes long before blocks. The error is the same ENOSPC, and no amount of deleting large files helps.

4. The everyday sweep

du -xh --max-depth=1 / 2>/dev/null | sort -h | tail -12
journalctl --disk-usage
sudo journalctl --vacuum-size=200M

-x keeps the walk on one filesystem, which is the flag that stops the search wandering into a network mount and taking twenty minutes to say nothing.

← cd .. fris@linux:~/blog$ man storage