iSCSI and Multipath: Sessions, Paths and the Layers Between Them

iscsiadm -m discovery -t st -p 10.0.5.10
iscsiadm -m node --login
iscsiadm -m session                        # what is actually up

The initiator IQN matters more than people expect: after a rebuild with a regenerated IQN, the array still masks the old one. Compare cat /etc/iscsi/initiatorname.iscsi with the array's host definition.

lsblk -S                      # SCSI devices and their transport
dmesg -T | grep -i -E 'iscsi|sd ' | tail
rescan when LUNs changed on the array:
echo "- - -" > /sys/class/scsi_host/host2/scan

New LUN invisible after array-side mapping? The host rescan is mandatory; iSCSI does not poll. A LUN that appears with the wrong size after expansion needs echo 1 > /sys/block/sdX/device/rescan.

multipath -ll                 # the whole picture: map, paths, checker state
multipath -t > /etc/multipath.conf   # start from detected defaults
cat /sys/block/dm-3/dm/uuid   # which map is which dm device

• No map at all: the device is in the blacklist (multipath -B shows blacklist reasons) or multipathd is not running. Arrays whose vendor string is unknown need an explicit devices section.
• Map exists with one path: the second session is down (back to layer one) or the second portal's path is faulty — multipath -ll shows each path with its checker result.

Paths oscillating failed/ready every few seconds usually mean the path checker is timing out on a congested, not dead, path — or a switch port with CRC errors. Evidence:

multipathd show paths          # live checker results
dmesg -T | grep -i 'error'     # I/O errors per sd device
ethtool -S eth2 | grep -i -E 'crc|drop'   # the network underneath

Response by cause: fix the physical path first; if the array is merely slow under load, tune the checker (path_checker tur vs directio) and consider no_path_retry queue so brief all-path blips queue I/O instead of erroring it. Never set failback and priorities blindly — a mis-set group-by policy can prefer the "wrong" controller and recreate the flapping as load imbalance.

When all paths of a mounted map die and return, the filesystem may have remounted read-only. Check mount | grep ro, and dmesg for "Remounting filesystem read-only". A clean path recovery still leaves you to remount rw (or reboot, for some stacks) — and to find out why the paths died, because the application just felt a storage outage, not a blip.

Multipath device names (/dev/mapper/mpatha) are assigned in discovery order and can shuffle after a reboot — an fstab on mpatha is a future wrong-filesystem mount. Use the stable paths: /dev/disk/by-id/dm-uuid-mpath-<WWID> or /dev/mapper/<wwid>, or define aliases in multipath.conf (alias db1 <wwid>) and mount by alias. After editing the config, reload with multipathd reconfigure (or multipath -r) and confirm multipath -ll reflects it. For partitioned LUNs, kpartx maps the partitions; a missing partition device after a resize is usually "kpartx never reran", fixed by re-adding the mapping rather than by rebooting.

Alert on path-count per map (multipath -ll parsed, or the exporter if you run one) dropping below the designed count, on iscsi session count changes, and on a canary file written through each mounted LUN every few minutes. The canary turns "paths flapped at 3 a.m. and nobody noticed" into a page with a timestamp, and the timestamp is half the root-cause work.

• Monitor multipath -ll path counts; alert on any map below its expected path count.
• Keep initiator IQNs in your provisioning records; array masking diffs are silent until a host rebuild.
• Log the network under the storage: switch port CRC counters catch dying cables weeks before outages.
• Test one deliberate path failover per maintenance window; untested multipath is a hope, not a design.