ESXi Purple Screen: Reading a PSOD Like a Crash Report
A PSOD is the vmkernel's last word: an exception or assertion it could not survive. The screen, the vmkernel log's final lines and the core dump together usually name the failing driver — and the driver names the fix.
Capture before you reboot
- Photo of the screen: the exception type, the backtrace (module names on the stack), and the register block all matter.
- If the host answers, or after reboot: the last lines of
/var/log/vmkernel.logon the persistent volume repeat the backtrace. - Note the ESXi build number — driver bugs are build-specific.
grep -i -B2 -A40 'Panic\|PF Exception\|ASSERT' /var/log/vmkernel.log | tail -80
Reading the backtrace: the module is the suspect
The stack lists VIB/driver names. The topmost non-vmkernel module at the time of the crash is your prime suspect:
| Module family on the stack | Usual story |
|---|---|
nmisc / net* (e1000, igbn, nmlx5...) | NIC driver/firmware bug or bad DMA — check the vendor driver vs the inbox one |
nvmw / pvscsi / nmlx4/nmlx5 (storage) | Storage HBA driver or firmware; also cables/SFPs producing storms |
vmklinux / hardware health | Often memory or CPU hardware error surfacing as an exception — check the IPMI/SEL log |
ASSERT ... vmkernel with a named source file | Kernel assertion; match build+assertion against known issues for your exact ESXi build |
The disciplined move: take "driver X, build Y, exception Z" to the vendor (Broadcom/VMware) knowledge base and to the NIC/HBA vendor's release notes. Most recurring PSODs have a published driver/firmware pairing fix.
The hardware layer: do not skip it
# after reboot, look for machine-check and hardware complaints: grep -i -E 'MCE|machine check|hardware|ecc' /var/log/vmkernel.log | tail esxcli hardware ipmi sel list 2>/dev/null || # via iLO/iDRAC instead: # read the server's SEL/event log for memory/CPU/PCIe AER entries
- Correctable-ECC storms that precede PSODs point at a DIMM; the SEL timestamps correlate with the crashes.
- PCIe AER errors correlate with a slot: reseat/swap the card before replacing software.
- PSODs under heavy network load with a specific NIC model = driver/firmware until proven otherwise.
Core dumps: when you need the full picture
If the host is configured with a dump partition or network collector (esxcli system coredump network get), the core lets support decode the full stack and data structures. Confirm collection is on before the next incident, not during it:
esxcli system coredump file get esxcli system coredump network get # analysis tooling (offline, on a support host): vsish / crash utilities via vendor support
For most teams the practical rule: capture screen+logs, correlate the module, apply the vendor's driver/firmware pairing; escalate with the core only when it recurs.
Post-PSOD checklist before returning to production
- Update the implicated driver/firmware per vendor matrix (or roll back a recent update that introduced it).
- Clear/verify hardware SEL; run the vendor diagnostics on memory and the suspect slot.
- Confirm the host rejoins the cluster with clean health alarms; watch it for 48 h before loading it back to full.
- If two hosts PSOD with the same stack in a window, treat it as a fleet issue: same driver version is the common factor; fix the baseline image, not the host.
No purple screen at all: the hung host
Some storage lockups freeze a host with no PSOD: management dies, VMs stall, the console shows a frozen frame. Configure the escape hatches before you need them: a network core-dump collector plus the ability to trigger an NMI from the BMC (iLO/iDRAC "power diag/NMI") — the NMI forces the vmkernel to dump and reboot, converting a mystery hang into an analyzable core with timestamps. Correlate the freeze time with the storage array's own event log first: a controller failover with broken ALUA or a dead path set is the most common author, and the vmkernel log's last lines (endless command aborts for one device) will say so. A host that hangs twice against the same array event is an array incident wearing a hypervisor costume.
What to attach when you escalate
Vendor cases move at the speed of their inputs. Attach: the PSOD photo or vmkernel backtrace text, exact ESXi build, the implicated driver's VIB version (esxcli software vib list), NIC/HBA firmware versions, the server model and the BMC SEL extract, plus crash timestamp and load conditions. "Host purple-screened, please advise" with those attachments gets a firmware-matrix answer; without them it gets a question-mark loop.
Fleet discipline: baseline the driver/firmware matrix
Keep an inventory of the crash-relevant versions so a PSOD is a lookup, not an investigation:
esxcli software vib list | grep -iE 'net|fc|nvme|scsi' esxcli network nic list # driver+firmware per NIC esxcli storage core adapter list # storage HBAs
Standardise on the vendor-supported driver/firmware pairing in the golden image; the recurring PSOD almost always lives in a host that drifted from the matrix (a one-off VIB install "to fix something" months earlier). Drift detection is one scheduled diff of the above output against the baseline.