Linux Won't Boot: Reading the Stage That Failed

grub rescue> ls
(hd0) (hd0,gpt2) (hd0,gpt1)
grub rescue> ls (hd0,gpt2)/boot/grub   # find where the modules live

Causes: a disk order change, a wiped /boot, or a clone that did not reinstall GRUB. Boot manually and then repair properly:

grub rescue> set prefix=(hd0,gpt2)/boot/grub
grub rescue> set root=(hd0,gpt2)
grub rescue> insmod normal
grub rescue> normal

Once up, reinstall from the running system: grub2-install /dev/sda && grub2-mkconfig -o /boot/grub2/grub.cfg (paths vary by distro — UEFI systems install to the ESP, not the MBR).

The initramfs's only job is to assemble and mount root. A dropped shell means it could not: missing driver (storage controller module not in the image), missing root device (UUID changed after cloning or disk swap), or LVM/RAID not activated.

# at the (initramfs) prompt:
blkid                          # what actually exists?
cat /proc/partitions
ls /dev/mapper                 # are LVM LVs visible?
lvm vgchange -ay               # activate them manually

The classic author: /etc/fstab or the kernel command line references a UUID that no longer exists. Compare blkid output with cat /proc/cmdline. Boot with a corrected root= from the GRUB menu (press e, edit, Ctrl-x), then fix fstab and rebuild the initramfs once up: dracut -f / update-initramfs -u.

systemd made it to userspace but a required unit failed: almost always an fstab entry that cannot mount (a removed data disk, a wrong NFS option) with the default nofail absent. The console usually says which mount. Repair:

# at the emergency shell (root password):
mount -o remount,rw /
systemctl --failed
blkid; vim /etc/fstab        # add nofail or correct the UUID
systemctl daemon-reload; reboot

Rule for shared/data entries you can live without at boot: nofail,x-systemd.automount — the boot never dies on a missing share again.

# from a rescue ISO:
mount /dev/sda2 /mnt
mount /dev/sda1 /mnt/boot/efi        # ESP on UEFI
for d in dev proc sys run; do mount --bind /$d /mnt/$d; done
chroot /mnt
grub2-install /dev/sda; grub2-mkconfig -o /boot/grub2/grub.cfg
dracut -f                              # rebuild initramfs while you are in there

Verify inside the chroot that /etc/fstab matches blkid before exiting — a repaired GRUB with a broken fstab just moves you to stage 3.

On UEFI systems the boot order lives in NVRAM, and firmware updates, battery-dead resets or cloned disks can drop the entry — the box then boots whatever is next (often a network PXE or the wrong disk) with GRUB perfectly healthy. Inspect and repair from a rescue environment:

efibootmgr -v                     # what the firmware remembers
efibootmgr --create --disk /dev/sda --part 1 \
  --loader /EFI/distro/shimx64.efi --label "Linux"
# universal fallback: copy the shim to the default path
cp /EFI/distro/shimx64.efi /EFI/BOOT/BOOTX64.EFI   # on the ESP

Also check ESP headroom: a full ESP (accumulated kernel/initramfs copies) makes GRUB and kernel updates fail mid-write — boot-breaking updates are preventable with a housekeeping cron for /boot/efi.

• /boot on RAID1: GRUB must be installed to each member disk; installing to one leaves the second disk unbootable the day the first dies — verify with mdadm --detail plus a deliberate BIOS boot order test.
• Saved-entry surprises: grub2-editenv list shows a persisted saved_entry that reboots into the old kernel "for no reason" after upgrades.
• Headless consoles: a kernel line missing console=ttyS0 makes a perfectly booting system look dead over serial — the stage-3 emergency shell may be printing somewhere you are not watching.

• Use UUIDs (or /dev/disk/by-*) everywhere; never raw device names that renumber.
• nofail on every non-essential fstab entry; boot resilience is a one-word change.
• After kernel or storage changes, keep the previous kernel entry; it is the fastest rollback.
• On VMs, exclude host-specific drivers assumptions: a cloned VM with the wrong disk controller in initramfs is stage 2 waiting to happen.