Ansible · Troubleshooting guide

Ansible Playbook Failures: Unreachable Hosts, Failed Tasks and Idempotency

Ansible errors fall into three layers: connection (the host is UNREACHABLE), execution (the task FAILED), and design (the playbook "works" but changes things every run). Each layer has its own debugging tools.

Published August 29, 2026 · RCW IT Training

Layer 1: UNREACHABLE — the connection never happened

Read the exact SSH reason.

ansible all -m ping -vvv
# common failures:
# ssh: connect to host 10.0.2.15 port 22: Connection refused
# Permission denied (publickey)
# Host key verification failed
# Failed to connect to the host via smart connection... timed out

Match the line to the cause: refused = sshd down or firewalled; permission denied = wrong key or user; host key verification = the host's key changed or was never accepted (set host_key_checking deliberately, don't just disable it); timed out = routing/security group.

Reproduce outside Ansible first.

ssh -o BatchMode=yes -o ConnectTimeout=5 ansible@10.0.2.15 'true' && echo OK

If plain SSH fails, no inventory change will fix it. Ansible uses your SSH layer; fix SSH, then rerun.

Inventory and user mismatches.

Playbook expects ansible_user=ansible but the inventory still says root — check with ansible-inventory --list.
Python missing on the target: /usr/bin/python3: No such file or directory — set ansible_python_interpreter=/usr/bin/python3 or bootstrap the package first.
Privilege escalation: "sudo: a password is required" — the become user needs NOPASSWD for the commands, or supply --ask-become-pass.

Layer 2: FAILED — the task ran and reported an error

Get the real module output.

ansible-playbook site.yml -vvv            # full module args + return JSON
ansible-playbook site.yml --start-at-task="Install packages"
ansible-playbook site.yml -l web2          # limit to one host while debugging

The -vvv JSON contains the module's msg, rc and stderr — the actual error is in there, not in the one-line summary.

The failures that repeat in every codebase.

Error fragment	Cause and fix
`non-zero return code` from a command that "succeeds"	The command exits non-zero by design (grep no match, systemctl is-enabled); use `failed_when` or `register` + `changed_when`
`Unable to find required dependency`	Module needs a library on the target (e.g. python3-pip, python3-firewall) — install it in a prior task
`Destination directory does not exist`	copy/template before the parent dir task ran; order or `directory: file` module first
Handler did not run	Handler names are unique per play; a notify typo fails silently at runtime — lint catches it
`MODULE FAILURE: SyntaxError`	Old Python on target vs new module syntax; pin `ansible_python_interpreter`

Layer 3: not idempotent — "changed" on every run

A playbook that reports changed every time is usually using command/shell where a declarative module exists. The audit is one flag:

ansible-playbook site.yml | grep changed=

Replace the usual offenders: shell: echo X > /etc/file → template/copy; shell: systemctl restart x → systemd: state=started enabled=yes plus a handler for config changes; shell: useradd → user module. Where a shell task is genuinely needed, teach it: creates=/path, removes=/path, or an explicit changed_when.

Vault, facts and variable surprises

Attempting to decrypt but no vault secret found — run with --vault-id or the env var; CI often lost the secret rotation.
Variables "disappearing": precedence bites — play vars beat role defaults; extra_vars beat everything. ansible -m debug -a "var=x" shows what actually resolves.
Facts slow or failing on minimal images: gather only what you need with gather_subset, or gather_facts: false when unused.

Resuming after failure: retry files and limits

When a 200-host run fails on five hosts, never rerun the whole fleet. Ansible writes a retry file listing the failures; combine it with --limit:

ansible-playbook site.yml --limit @site.retry
ansible-playbook site.yml --limit 'web*'        # pattern-based slice
ansible-playbook site.yml --step                # confirm each task, for fragile paths

--step is the underused safety net for running an unfamiliar playbook against production: you approve task by task and can bail before the destructive one.

Speed knobs that also change failure shapes.

pipelining = True removes per-task SSH round trips; if tasks suddenly fail with sudo errors after enabling it, the target's requiretty sudoers setting is the culprit.
forks controls parallelism; a run that "worked in dev" and fails in prod at 200 hosts is often your control node or a network appliance rate-limiting SSH — raise forks gradually and watch both ends.
any_errors_fatal vs max_fail_percentage: choose explicitly how a fleet run aborts, or the default (keep going, report at end) will surprise someone at 2 a.m.

Prevention checklist

Run ansible-lint and ansible-playbook --syntax-check in CI before anything touches hosts.
Dry-run with --check --diff on every change; diff output is what makes reviews meaningful.
Keep a bootstrap play (raw/ssh-only) separate, so new hosts with no Python don't break the main playbook.
Limit blast radius in prod: -l one_host, then serial: 25% for fleets.

Key takeaway: Debug Ansible in layers: reproduce UNREACHABLE with plain SSH, read the module JSON with -vvv for FAILED tasks, and use --check plus declarative modules to kill "changed on every run". The tool is honest; the error text, expanded with verbosity, names the layer.