Linux and RHEL · Troubleshooting guide

Linux Troubleshooting: Boot Failures, Read-Only Roots and Phantom Disk Usage

Field-tested fixes for the three calls every Linux admin gets: a system that will not boot, a root filesystem that flipped read-only, and a disk that is full but shows no files.

Published August 28, 2026 · RCW IT Training

Boot and filesystem emergencies

GRUB drops to a rescue shell after a kernel or package update

Usually a missing or mismatched initramfs, or a wrong root= UUID. Boot the previous kernel from the GRUB menu; if that fails, chroot from a rescue ISO: mount the root and /boot, mount --bind /dev /proc /sys, then chroot /mnt/sysimage. Rebuild the initramfs with dracut -f, regenerate the menu with grub2-mkconfig -o /boot/grub2/grub.cfg, and confirm every /etc/fstab UUID still matches blkid output before rebooting.

Root filesystem remounts read-only in the middle of work

The kernel flips a disk read-only when it sees I/O errors - treat this as a storage alert, not a Linux bug. Check dmesg -T | grep -i error and smartctl -a /dev/sdX. If the hardware is healthy, run fsck from rescue mode (never on a mounted filesystem), then mount -o remount,rw /. On cloud VMs also check the hypervisor or provider event log for datastore incidents.

df says 100% full but du cannot find the files

Classic deleted-but-still-open files: a process keeps writing to an unlinked file, so space is allocated but invisible. List offenders with lsof +L1 | sort -k7 -rn, then restart the service or truncate the handle with > /proc/<pid>/fd/<n>. Long term, ship logs to a collector and alert on both df usage and deleted-inode space.

Prevention checklist

Make boots boring

Keep two known-good kernels, test updates on a snapshot first, and verify grubby --default-kernel after every kernel install. Reboot in a scheduled window, never as a surprise.

Alert on the right signals

Monitor dmesg I/O errors, SMART reallocated sector counts, df above 80%, and deleted-inode space. A read-only remount should page you before users notice it.

Key takeaway: Boot failures are usually UUID or initramfs drift, read-only roots are the storage layer talking to you, and 'full but empty' disks are open deleted files - run lsof +L1 before deleting anything.