rsync and restic Failures: Exit Codes, Locks and the Restore You Never Tested

Exit 23 with Permission denied lines: run as root, or accept it consciously with --super/--no-perms semantics understood — silently skipping unreadable files is how backups lose exactly the sensitive directories.

rsync -a /data backup:/bk/ copies the directory; rsync -a /data/ backup:/bk/ copies its contents. The wrong variant produces a backup that restores to the wrong shape — a success that fails at restore. Standardise one form in your scripts and document it. Related traps: forgetting -a (no recursion/permissions), and --delete pointed at the wrong target — test a new rsync line once with -n before scheduling it.

restic -r /repo snapshots            # is the repo readable at all?
restic -r /repo check --read-data-subset=5%   # real integrity sampling
restic unlock                        # ONLY after confirming no backup is running

• "repository is locked": a previous run died (OOM, Ctrl-C, host crash) leaving a stale lock. Confirm nothing is running on any host sharing the repo, then restic unlock. Never unlock blindly on a shared repo.
• OOM during backup of huge trees: restic scans and packs in memory; limit with --pack-size considerations or exclude monster directories; the process dying silently mid-run is the classic "no snapshot since Tuesday".
• check failures: run restic repair index first; data-loss-level corruption means restore what verifies from older snapshots — another argument for check in the schedule, not just backup.

The most common backup outage is a cron that stopped: a changed passwordless-ssh key, a moved repo mount, or the job waiting forever on a hung NFS source. Detect it from the destination side — a freshness check beats reading cron mail:

# on the backup target or a monitor:
find /backups/host1 -maxdepth 1 -mtime -2 | grep . || echo "NO FRESH BACKUP"

Alert on snapshot age per host, and on the job's own runtime doubling (a backup that suddenly takes 10× is reading a broken disk or lost its incremental state).

• Weekly: restore a random snapshot to scratch space and diff -r a sample tree; log the result.
• Quarterly: full restore drill of one critical dataset, timed, with the runbook in hand.
• For restic: restic restore --target /drill <snap> --include /critical/path; time it — your RTO is the measured number, not the hoped one.
• After every tool or OS upgrade on the backup host, run one drill before trusting the next scheduled run.

A restic repository without its key is noise; a key without discipline is a single-file loss of every backup. Keep at least one recovery copy of the key offline and separate from the repository (a different vault, a sealed-envelope process, a secrets manager with its own access review), and prove the copy works: a quarterly drill that restores using the recovery key, not the one cached on the backup host. restic key list and restic key add let several keys protect one repo — issue a dedicated key to the drill so it never touches production credentials. And keep the threat model split in view: object lock makes backups undeletable, encryption makes them unreadable — you need both properties, and "stolen bucket plus leaked key" is the scenario that tests whether key storage was real.