NFS and SMB Mount Failures: Stale Handles, Permission Denied and Hung Mounts

showmount -e nfs.example.com          # is the export list reachable?
rpcinfo -p nfs.example.com            # are rpc services answering?
mount -v -t nfs nfs.example.com:/data /mnt/data

showmount failing with "RPC: Program not registered" means the NFS server service or exports are not loaded (systemctl status nfs-server, exportfs -rav). "Connection refused" on port 2049 is a firewall or a server listening only on IPv6/another interface.

smbclient -L //fs1.example.com -U user%pass
smbclient //fs1.example.com/share -U user%pass -c 'ls'
# if names fail but IPs work:
nmblookup fs1

NT_STATUS_CONNECTION_REFUSED = smbd down or port 445 blocked; NT_STATUS_ACCESS_DENIED at -L = credentials or "restrict anonymous"; NT_STATUS_BAD_NETWORK_NAME = the share name does not exist on that server.

The client's UID/GID, the server's export options, and the server's filesystem ACL all get a vote. The classic traps:

• root_squash (default): root on the client becomes nobody on the server — root-owned writes fail even though the mount succeeded. Test as a normal user before blaming the share.
• UID mismatch: NFS trusts numeric IDs. User 1001 on the client is whoever 1001 is on the server. Align IDs (SSSD/FreeIPA/LDAP) or map them.
• SELinux on the server: exports from non-standard paths need the right context (semanage fcontext, or use /srv conventions) — check ausearch -m avc.

# server side, after changing /etc/exports:
exportfs -rav
# client side, who am I really?
sudo -u appuser touch /mnt/data/.w && echo ok

Samba evaluates its own share ACLs (valid users, write list, read only) and the underlying filesystem permissions; the stricter wins. On SELinux hosts, shared paths need the samba_share_t context or the samba_export_all_ro/rw boolean. And remember Windows-side "Map network drive" caches credentials — a password change produces "access denied" until the old session is dropped (net use * /delete).

Stale file handle (NFS) means the client holds a file handle the server no longer honours: the export was recreated, the backing filesystem re-created or re-imported, or the server rebooted with a different fsid. Immediate relief is a remount; the fix is process — never delete and recreate an exported directory while clients mount it.

umount -l /mnt/data        # lazy unmount if busy
mount -t nfs nfs.example.com:/data /mnt/data
# find processes holding it:
fuser -vm /mnt/data

A hard-mounted NFS share with a dead server makes every touching process sleep in state D — uninterruptible, not killable. This is why production fstabs use resilient options:

server:/data  /mnt/data  nfs  defaults,_netdev,nofail,soft,timeo=100,retrans=3  0 0

• _netdev + nofail: boot continues when the network or server is slow — the classic "server rebooted and three VMs now hang at boot" is a missing nofail.
• soft returns errors instead of hanging forever; choose it for non-critical shares, hard for databases and logs where silent data loss is worse than a hang.
• For SMB: vers=3 or higher, and credentials= file instead of inline passwords.

When exports use Kerberos, a whole extra failure layer appears between "network ok" and "mount ok": expired or missing host keytabs, clock skew beyond the five-minute Kerberos window, and mismatched security flavours (client mounts sys against a krb5p-only export and gets access denied by server). Diagnose in order: klist -k on the client for the machine principal, date synchronisation against the KDC, then the exact sec= flavour on both sides. The server log line RPC: AUTH_ERROR: bad credential points at the keytab/clock pair; wrong security flavor points at the export options.

Two recurring incidents: shares that "move" because clients actually connect a DFS namespace path — troubleshoot the namespace targets, not the server you assume; and signing mismatches, where a hardened client requiring SMB signing meets a server with it disabled (or the reverse after a hardening project), producing access errors that look like permission problems. Compare Get-SmbClientConfiguration with the server's signing policy before touching share ACLs.

• Put _netdev,nofail on every network fstab entry; boot-hangs are fully preventable.
• Document UID/GID strategy per share; NFS permission bugs are identity bugs.
• Monitor mount liveness from the client side (touch a canary file) — server health checks miss export-level breaks.
• Change exports with exportfs -r and never recreate exported directories in place.
• For SMB, disable SMB1 everywhere; it causes both failures and security findings.