ZFS Pool Degraded: Reading status, Choosing Clear or Replace

zpool status -v POOL
zpool status -x              # every pool with problems
# the columns that matter:
#   STATE: ONLINE / DEGRADED / FAULTED / SUSPENDED
#   READ WRITE CKSUM errors per device
#   the action: line — ZFS's own recommendation

• DEGRADED: a device is offline/unavailable but redundancy covers it. Data is safe while the remaining mirrors/parity hold.
• FAULTED: the device (or enough devices) failed such that ZFS cannot guarantee data; on a pool without redundancy this is data loss.
• Checksum errors on an ONLINE device: the disk is lying to you — bad sectors, a dying cable, or a controller writing garbage. Small counts that stop may be cabling; growing counts are a disk on its way out.

If a device went OFFLINE after a transient event (cable bump, enclosure reset, one bad night) and now shows zero error growth:

zpool clear POOL DEVICE     # acknowledge errors, let it rejoin
zpool status -v POOL        # watch for resilver or error regrowth

Replace instead when: errors keep growing, the disk is slow/unresponsive (SMART reallocated sectors, long read times), or the pool lost redundancy and you want zero exposure window:

zpool replace POOL OLDDEV NEWDEV
zpool status -v POOL        # resilver progress under 'scan:'

Never clear a device with climbing CKSUM counts just to make the alert go away — you would be re-admitting a corrupting device.

zpool scrub POOL
zpool status -v POOL | grep scan     # progress + errors found
zpool history POOL                    # what ran when; great for postmortems

• A scrub after any disk event is non-negotiable: it verifies the redundancy actually reconstructs.
• Scrub errors found and repaired with all devices online = healed bit rot; note it and move on. Errors that cannot be repaired (no good copy) mean damaged files — zpool status -v lists them; restore those files from backup.
• During a resilver the pool is one more disk failure from data loss on RAID-Z1/mirror-minus-one: schedule heavy work elsewhere and replace promptly.

A disk that has not failed but responds in seconds stalls the whole vdev — users feel it as freezes. Evidence: iostat with one device's wait far above its siblings, dmesg with repeated reset/timeout lines for one disk. ZFS has no mercy for latency; the fix is replacement (or, as a stopgap, zpool offline it if redundancy allows, converting latency into a clean degraded state).

zpool iostat -w 5 POOL      # per-device wait times expose the liar
dmesg -T | grep -i -E 'reset|timeout|error' | tail

zpool import              # lists importable pools, including FAULTED ones
zpool import -f POOL      # after unclean export or moved devices
zpool import -m POOL      # import missing log device (may lose last sync txgs)

Before forcing: check dmesg that all member disks actually appeared (a missing HBA shows up as "pool was previously imported but devices are missing"). Importing degraded on purpose beats not importing at all when the alternative is waiting for hardware at 2 a.m. — but log what you forced, for the postmortem.

Resilver time is data volume divided by achievable rebuild rate, and real-world rates on large spinning pools are humbling — a 40 TB RAID-Z2 can take a day or more, during which the pool runs one fault thinner. Plan for that: add a hot spare device (zpool add pool spare DEV) so ZFS promotes it automatically when a member faults; know your scrub rate (time the monthly scrub) as your resilver estimate; and never start capacity changes or heavy scrubs on a degraded pool. Mirror vdevs resilver far faster than RAID-Z — a legitimate design argument for mirrors on pools where exposure time matters more than raw capacity per disk.

• Scrub monthly (weekly on big slow pools); the point is finding rot while redundancy still heals it.
• Alert on any device with nonzero and growing READ/WRITE/CKSUM, and on pool state != ONLINE.
• Keep spares and a documented replace runbook; resilver time is your real exposure window.
• Log zpool history to your central logs — postmortems become trivial.