Google Cloud · Troubleshooting guide

Cannot SSH to a GCP VM: A Layer-by-Layer Diagnosis

SSH failures in Google Cloud are never mysterious — they live in one of four layers: the network path, the firewall, the guest OS, or your identity. Check them in that order and each failure message maps to exactly one.

Published August 29, 2026 · RCW IT Training

Read the client error first

Client message	Layer
`Connection timed out`	Firewall drop or no route — packets never reach sshd
`Connection refused`	VM is up but sshd is not listening (or not on 22)
`Permission denied (publickey)`	Identity/keys layer — you reached sshd fine
Console shows "Waiting for SSH" forever	Guest never finished booting — OS layer

Timeout and refused are different universes: one says the packet died en route, the other says the OS answered. Never debug keys on a timeout.

Layer 1–2: network path and firewall

gcloud compute firewall-rules list \
  --filter="direction=INGRESS" --format="table(name,disabled,sourceRanges,allowed)"
gcloud compute instances describe VM --zone=Z \
  --format="value(networkInterfaces[0].accessConfigs[0].natIP)"

No external IP? The Console SSH button and your laptop both need one (or IAP/Identity-Aware Proxy tunneling, or gcloud compute ssh with an internal-only setup uses the API tunnel — check which path you are on).
The default default-allow-ssh rule covers tagged/whole-network VMs; a custom VPC created without it silently times out every SSH.
Source range: the rule may allow only your office IP; a changed egress IP becomes "SSH broke" the next morning.

Layer 3: the guest OS

When the network looks right, go through the serial console — it works even when SSH is dead:

# Console → VM instance details → Serial port 1 (interactive), or:
gcloud compute connect-to-serial-port VM --zone=Z

What to look for: kernel panic or emergency target (boot failure, often a bad /etc/fstab entry added before reboot), sshd not running (systemctl status sshd), or a full disk (df -h — a 100% root filesystem stops sshd accepting sessions in odd ways). A VM that "refuses" after a package upgrade frequently has sshd crashed on a bad config; sshd -t inside the serial console names the line.

Layer 4: keys, OS Login and the account you think you are

Two identity systems coexist: project/instance metadata keys and OS Login. If OS Login is enabled on the project, metadata keys are ignored — the classic "I added my key, still denied" incident. Check:

gcloud compute instances describe VM --zone=Z \
  --format="value(metadata.items[enable-oslogin].value)"
gcloud compute os-login describe-profile

Also verify the username: connecting as ubuntu to a Debian image, or as your personal OS Login user while the key sits under another account's metadata, both end in Permission denied (publickey). Audit what the instance actually holds: gcloud compute instances describe --format=yaml | grep -A5 ssh-keys.

When the VM itself will not boot

Serial log stuck before login prompt → boot disk issue; detach and attach it to a debug instance to repair fstab/grub.
gcloud compute instances get-serial-port-output gives you the last boot even without interactive access.
Startup scripts that hang (waiting on a dead NFS mount) delay sshd for minutes — the serial output shows the script, not a crash.

When your SSH works but the Console "Connect" button fails

The browser SSH button is a different path from your terminal: it tunnels through the API (or IAP) and authenticates with OS Login or pushed metadata keys. A 403 on the button with a working CLI usually means your account lacks the OS Login role or compute.instances.setMetadata; "connection failed" with a working CLI usually means the guest agent that accepts those keys is not running. Check from the serial console: systemctl status google-oslogin-service (and the sshd PAM wiring for OS Login). Also mind the firewall direction: console/IAP traffic arrives from Google's IAP range (35.235.192.0/20), not your office IP — a firewall that whitelists "the admins" and not that range produces exactly this split-brain symptom.

One-minute triage table

Symptom	Layer	First command
timeout	firewall/route	`gcloud compute firewall-rules list --filter=direction=INGRESS`
refused	sshd	serial console: `systemctl status sshd; sshd -t`
permission denied	identity	`gcloud compute os-login describe-profile`
waiting for SSH	boot	`gcloud compute instances get-serial-port-output`

Evidence to collect while it is broken

Grab, in one place: the exact client error text, gcloud compute instances get-serial-port-output, the effective firewall rules for the instance's network and tags (gcloud compute firewall-rules list --format=yaml filtered), VPC Flow Logs rows for port 22 during the attempt (they show DROP with the rule that matched — the single fastest firewall answer), and the identity you connected as. With those five artifacts, every layer is either proven or excluded, and the incident note writes itself.

Prevention checklist

Prefer IAP TCP forwarding plus OS Login over public IPs and metadata keys — fewer layers, better audit.
Keep one tagged "break-glass" firewall rule and one serial-console-capable admin path documented.
Alert on boot-disk usage; a full root disk is the most common silent sshd killer.
After image or package upgrades, verify sshd -t passes in your golden-image pipeline, not in production.

Key takeaway: Classify by the client error first — timeout is firewall/route, refused is sshd, permission denied is identity, and "waiting for SSH" is the guest. The serial console is the universal fallback that turns a dead-VM mystery into a readable log.