"No Space Left on Device" with Free Space: The Four Causes

Applications fail on a filesystem you did not check: /var, /tmp, or a separate /var/log at 100% while / sits at 40%. Check them all, and check what the failing service actually uses:

df -h
df -h /var /tmp /var/log
# which filesystem owns the failing path?
df -h /var/lib/mysql/

Also remember bind mounts and containers: a container's overlay can be full while the host looks fine — docker system df is the container-world equivalent of df.

ext4 allocates a fixed inode count at mkfs time. A filesystem can run out of inodes at 10% block usage: session files, cache shards, mail queues, millions of tiny logs. Proof:

df -i
Filesystem      Inodes  IUsed IFree IUse% Mounted on
/dev/sda1      6553600 6553598     2  100% /

Find the directory with the horde and delete in batches (a bare rm * on a million files hits ARG_MAX and hangs your shell):

for d in /var/spool/*/; do echo "$d $(ls -f "$d" | wc -l)"; done | sort -k2 -n | tail
find /var/cache/app -type f -delete        # kernel-side, no arg limit
# fast bulk empty of one dir:
rsync -a --delete /empty/ /var/cache/app/

ext4 keeps ~5% reserved for root so a full disk never kills the OS. On a 2 TB data volume that "protection" is 100 GB of space a service account sees as No space left at 95% usage. Proof: df shows 95% used and the failure is for a non-root user. Adjust deliberately:

tune2fs -l /dev/sdb1 | grep -i 'reserved block count'
tune2fs -m 1 /dev/sdb1     # 1% is plenty for data volumes

The signature mismatch: df 100%, du -sh / far smaller. A process holds an unlinked file open — typically a log that was "deleted" without restarting the daemon that writes it. The space returns only when the fd closes:

lsof +L1 | sort -k7 -n | tail     # deleted files with open handles
# free it without downtime:
: > /proc/<PID>/fd/<FD>          # truncate through the handle
# or restart the owning service

This is why logrotate uses copytruncate or a proper reload signal — "rm the log to fix the disk" is exactly how you create this state.

du -xh / 2>/dev/null | sort -rh | head -15     # -x: stay on this filesystem
ncdu -x /                                      # interactive, if available
# per-directory inode counts when du looks innocent:
find / -xdev -type d -exec sh -c 'echo "$(ls -f "$1" | wc -l) $1"' _ {} \; | sort -rn | head

/run, /tmp and /dev/shm are memory-backed; when they fill, failures look bizarre — services crash-looping, login shells misbehaving, systemd units failing with "No space left" while every real filesystem shows headroom. df -h -t tmpfs isolates them in one line. The systemd journal is its own serial offender on small /var: check journalctl --disk-usage and cap it with journalctl --vacuum-size=200M plus SystemMaxUse in journald.conf. Docker hosts add another hidden tenant: /var/lib/docker as its own mount fills from build caches and dead containers — docker system df then docker system prune with intent, not panic.

On multi-user systems, repquota -aus can show a user at their block or inode limit while the filesystem is half empty — the service account writing "No space left" is quota-bound, not disk-bound. The df/du pair never reveals this; only quota tooling does. If quotas are in play, include them in the standard four-way check so the diagnosis stays one command deep.

The kernel says it plainly when writes fail: dmesg -T | grep -i 'No space' names the device at the moment of failure — useful when the app log only shows a generic I/O error. Then sweep for the owners: df -Th (the -T exposes a surprise filesystem type like an unmonitored tmpfs or overlay), and find / -xdev -xtype f -size +500M -exec ls -lh {} + for the few giants that usually explain a jump. Five-hundred-megabyte files and million-file directories are the two shapes; the tools above name which one you have in under a minute.

• Alert on both block and inode percentage; inode alerts fire first on cache-heavy boxes.
• Never rm an active log; rotate it properly.
• Set tune2fs -m 1 on large data filesystems at build time.
• Log-ship or cap /var/log; a silent logger is the top author of every one of the four causes.