Terraform State, Locks and Drift: Recovering a Broken Plan
State lock errors stop pipelines; drift turns plans into surprises; partial applies leave state half-written. This guide covers the recovery path for each — in the order you will meet them.
Error acquiring the state lock
Error: Error acquiring the state lock ... Lock Info: ID: 7f3a... Path: s3://bucket/env/prod/terraform.tfstate Who: runner@ci-node-3 Created: 2026-08-28 22:41:07 UTC
A lock means another run was (or died) in the middle of writing state. The fix order: first, find out whether a run is genuinely active — check your CI for a running pipeline from that node and time. If a run is live, wait. If the process is dead, the lock is stale.
terraform force-unlock <LOCK_ID>
Force-unlock touches only the lock, never the resources. The danger is not the command; it is unlocking while a live apply is still writing — two writers corrupt state. Confirm the pipeline is dead (and the Created timestamp is hours old) before unlocking.
Drift: reality no longer matches state
terraform plan proposes to "fix" resources nobody asked to change, or shows Objects have changed outside of Terraform notes. Causes: someone edited the resource in the console or CLI, an autoscaler or operator mutated it, or a previous apply partially succeeded.
terraform plan -refresh-only # show drift without proposing changes terraform refresh # (legacy) write drifted values into state
- Accept reality: the manual change is correct →
terraform apply -refresh-onlyrecords it in state; then move the change into code so the next plan is clean. - Revert reality: the manual change is wrong → a normal
terraform applyputs the declared configuration back. - Adopt the resource: it exists in the cloud but not in state (created outside Terraform) →
terraform import(or the import block) maps it to an address, then reconcile attributes in code.
Partial applies and tainted resources
If an apply fails halfway, Terraform has already written what it created. The failed resource may be marked tainted — created, but considered broken:
terraform state list terraform plan # shows "will be replaced" for tainted items # if the resource is actually fine: terraform untaint <address> # if replacement is right: terraform apply # replaces it
Never hand-edit the JSON state file to "fix" an apply failure unless a provider bug left an orphan; prefer terraform state rm + import, which keeps the backup mechanism intact.
Provider and API errors that look like state errors
| Error | Real cause |
|---|---|
ResourceNotFound during plan | Resource deleted outside Terraform; refresh marks it gone and plan proposes re-creation — usually correct, verify no dependency expects the old ID |
AccessDenied / 403 | Expired or rotated CI credentials, or a policy change; not a state problem at all |
timeout while waiting for state | Cloud-side eventual consistency or a stuck async operation; check the provider's console task before retrying |
state snapshot too old / checksum errors | Concurrent writers or a truncated upload — inspect the backend's version history and restore the last good version |
Backend hygiene that prevents most incidents
- Use S3 (or equivalent) with locking enabled (DynamoDB or native bucket locking) — local state makes every one of the above worse.
- Enable backend versioning so a bad state write is a one-click restore.
- One workspace/root per environment; shared roots multiply lock contention and blast radius.
- Alert on any pipeline that exits non-zero mid-apply; a half-finished apply is the parent of tomorrow's drift.
- Tag every cloud resource with
managed-by = terraformso manual editors think twice and drift reviews are quick.
Orphaned resources and the state rm discipline
An orphan is a resource that exists in state but no longer in the cloud, or the reverse. When a resource vanished outside Terraform, plan shows it as gone and apply cleans the state entry — usually painless. The painful direction is a cloud resource with no state entry: Terraform cannot see it, will happily create a conflicting duplicate, and your cost dashboard notices before your pipeline does. A quarterly terraform plan -refresh-only plus a cloud-inventory-vs-state diff (tools like driftctl-style scanners, or a simple resource-list compare) catches both directions.
terraform state list # what state believes # compare with the cloud's resource list for the same tags/prefix terraform state rm <address> # remove from state WITHOUT touching the cloud terraform import <address> <cloud-id> # adopt a cloud resource into state
Rule of thumb: state rm then import is the repair kit; hand-editing state JSON is the last resort, and only from a downloaded copy with the backend locked.
TF_LOG=DEBUG TF_LOG_PATH=/tmp/tf.log terraform plan
Provider-level logs show the actual API requests and responses — the difference between "Terraform says 403" and seeing the exact IAM action that was denied. Attach the relevant log excerpt, not a screenshot of the summary line, when escalating to provider issues.
A safe incident order
- Confirm no live run; then
force-unlockif stale. terraform plan -refresh-onlyto see pure drift.- Decide per resource: accept, revert, or import.
- Only then run the real plan/apply.
- Post-incident: add the missing lock/versioning/alert that let it happen.