Datastore Full: Snapshots, Thin Disks and the Space You Cannot See

A snapshot is not a backup; it is a redirect layer that grows with every write. A busy VM's snapshot can outgrow its own disk in days. Find the hogs:

# vCenter: VM → Snapshots → Manage, or powercli:
Get-VM | Get-Snapshot | Select VM,Name,Created,@{n="SizeGB";e={[math]::Round($_.SizeGB,1)}}

Consolidate, don't just delete blindly during production peaks: snapshot removal triggers a stun while delta disks merge. Watch the task's "consolidate" helper warnings — a VM that shows Virtual machine disks need consolidation has orphaned deltas eating space right now.

Thin disks let you promise 10 TB from a 4 TB datastore. The bill arrives when guests actually write: the datastore crosses 95%, vSphere alarms fire, and if the backing array is also thin and full, writes can fail across every VM at once. Check both layers:

# datastore level:
esxcli storage filesystem list
# array level: the storage team's capacity view — vCenter shows consumed,
# the array shows the truth of the pool

Set the datastore alarm at 80%, not 95%: the last 15% of a shared datastore disappears in hours during a snapshot storm or a patching window.

Deleting files inside a VM does not return space to a thin datastore unless UNMAP happens. Modern VMFS6 reclaims automatically but lazily; VMFS5 and some NFS arrays never do without a push:

esxcli storage vmfs unmap -l DATASTORE_NAME -n 200   # iterate until done
# guest-side (so the virtual disk can shrink afterwards):
fstrim -av        # in the Linux guest, if the vmdk is thin and controller says trim

Measure before and after with esxcli storage filesystem list — if UNMAP moves nothing, the array may not support it or the space is genuinely consumed (snapshots!).

• VMFS: grow the backing LUN on the array, then esxcli storage core device list rescan and extend the extent in the datastore properties — online.
• NFS: grow the export on the filer; the datastore follows, but ESXi may need a remount to see it.
• No room to grow? Storage vMotion the largest VMs to a sibling datastore first; capacity moves are online and buy hours.

At 100% a VMFS datastore refuses new allocations: VMs with thin disks can suspend or crash, and No space left on device appears in VM logs. Immediate order: (1) delete or consolidate the largest snapshot, (2) move or delete orphaned files (/vmfs/volumes/DS/ browse for folders with no VM inventory entry — leftovers of removed VMs are gigabytes of rent-free space), (3) extend. Orphan hunting: compare datastore browser contents with the inventory; leftovers of cold-migrated or failed-removal VMs are common.

On vSAN the datastore math differs: reserve the documented slack (plan around 25–30% free; rebuilds and component limits degrade sharply past it), and remember that dedupe/compression savings show as "used" differently than on VMFS — a "full" alarm right after enabling space efficiency is often accounting, not data. Capacity operations also move components, so a disk-group failure during a full period can cascade into an unmountable object set. Monitor vSAN's own health checks (the capacity-what-if in the UI answers "can I lose a disk group now?") and treat that answer as your real alarm threshold, because the plain free-space percentage understates the risk on a deduped, near-full cluster.

# compare inventory to datastore contents; leftovers = VMs removed badly
ls /vmfs/volumes/DS1/
# each folder should match a VM in the inventory; the rest are orphans.
# size them before deleting anything:
du -sh /vmfs/volumes/DS1/suspect-folder

Delete only after confirming no VM references the files (check snapshots and templates too) — an "orphan" that is a template's disk is an outage, not a cleanup.

Define, at minimum: datastore usage > 80% (warning) and > 90% (alert); any snapshot older than 72 h; and per-array, pool capacity > 80%. The snapshot alarm is the one that prevents the incident class entirely — most "datastore full at 2 a.m." stories begin as "backup job left its snapshot on Friday". Pair each alarm with the runbook line that resolves it (consolidate, extend, SvMotion) so the page is actionable rather than atmospheric.

• Alarm at 80% datastore usage and on any snapshot older than 72 h.
• Audit snapshots weekly; backup jobs that use snapshots must delete them even on failure.
• Track array pool capacity separately from datastore capacity — two thin layers multiply the surprise.
• Schedule UNMAP/fstrim where reclamation is manual; treat it as maintenance, not emergency.