Object-Store Backups: rclone and restic Failures Against S3-Compatible Targets

rclone sync /data remote:bucket/data -P --stats-one-line
# on failure, the log shows e.g.:
# ERROR : file.bin: Failed to copy: ... 429 Too Many Requests / 503 SlowDown

Providers rate-limit per bucket and per prefix. Remedies that actually work: enable the tool's backoff (--low-level-retries, --retries-sleep in rclone; restic retries internally), spread keys across prefixes instead of one giant flat folder, and cap parallelism (--transfers, --checkers) — more parallelism against a throttled bucket makes it worse. For very large seeds, initial sync off-peak; steady state never throttles.

"Worked for months, now every run fails" against object storage is usually identity, not data:

• Rotated access keys not updated in the backup host's config (rclone.conf, restic env, or the secret your CI injects).
• A policy change removing s3:PutObject or adding a condition (source IP, MFA) that the backup host fails.
• Endpoint moved (region change, provider migration) while the config still names the old one — errors look like auth failures because the wrong endpoint answers.

rclone lsd remote: --verbose     # cheapest end-to-end credential test
restic -r s3:endpoint/bucket snapshots   # same for restic

Put one of those lines in your monitoring as a daily canary — credential drift should page you on day one, not on restore day.

restic check --read-data-subset=10%
restic repair index          # when snapshots list is wrong/empty but data exists
restic prune                 # housekeeping; run it or the repo grows forever

• "load: invalid passphrase or corrupted data" after a config migration: wrong key supplied (environment precedence bites — RESTIC_PASSWORD_FILE vs RESTIC_PASSWORD).
• Missing blobs during restore: a partially uploaded pack from a killed run; repair index reconciles, then re-verify with check.
• Never let two hosts write the same restic repo concurrently — restic tolerates it less badly than it looks, and locks will stall both.

Compliance/retention locks (Object Lock, S3 Object Lock in compliance mode) make deletes fail with 403 until the retain date — including your retention script's "delete backups older than X days". The failure looks like broken credentials or a broken prune. Check the bucket's lock configuration and the objects' retain-until dates before debugging anything else:

rclone backend command remote:bucket versioning   # or the provider CLI:
aws s3api head-object --bucket b --key k --query 'ObjectLock*'

Design retention around the lock: pick the retention window deliberately (e.g. 90 days), and let locked objects expire by policy instead of by delete. Ransomware-proof and human-proof — including proof against your own cleanup job.

• Versioning-enabled buckets: a "deleted" backup may be a delete-marker; list versions before concluding data loss.
• Restore from Glacier-class storage needs a thaw (minutes to hours) — budget it in the RTO, not in the panic.
• rclone copy vs sync at restore: sync deletes local extras; on a drill restore into scratch, prefer copy.

Backups that share one egress link or one bucket throttle each other. Shape with intent — rclone's time-based limits keep business hours clean:

rclone sync ... --bwlimit "07:00 20M 19:00 200M 23:00 off"
# restic has no built-in schedule; own the window in cron and time it:
time restic backup /data --read-concurrency 2

Stagger host start times so N hosts do not open N parallel multipart storms at 01:00, and alert when a job overruns its window — an overrun is the earliest signal of dataset growth, throttling, or a disk reading slowly. The window is a capacity contract: measure achieved throughput monthly and resize the schedule before the schedule resizes itself at 3 a.m.

Backup hosts quietly upgrading restic/rclone mid-incident is a classic confounder: repo format features, default retry behaviour and S3 signing defaults all change across versions. Pin versions in your config management, and test upgrades against a scratch repository first — the tool that guards your data deserves the same change discipline as the data's owners.

• Daily canary: list the repo/bucket from the real backup identity; alert on failure.
• Keep keys in one secrets source; document precedence of env/file config.
• Run restic check with data sampling weekly; integrity you don't sample is assumed, not known.
• Model retention with locks on paper before enabling them; include the prune job's behaviour in the design.