Stuck GKE Upgrade: Reading the Operation and Unblocking It

gcloud container operations list --filter="status!=DONE" --format="table(name,type,status)"
gcloud container operations describe OP_NAME --format=yaml | sed -n '1,60p'
# cluster-level state:
gcloud container clusters describe CLUSTER --format="value(status,currentMasterVersion,statusMessage)"

A cluster stuck in RECONCILING is mid-operation; do not start another operation on top of it. The describe output's statusMessage and nested operationProgress name the blocking step (control plane, a specific node pool, or an addon).

Node upgrades drain nodes one surge at a time; a PodDisruptionBudget that cannot be satisfied stops the drain, and the upgrade sits at a percentage for hours. Find it:

kubectl get pdb -A
kubectl get events -A --field-selector reason=EvictionFailed 2>/dev/null
kubectl top nodes 2>/dev/null; kubectl get pods -A -o wide | grep -c NODE_TO_DRAIN

Typical culprits: minAvailable: 100%-style PDBs, a single-replica Deployment with a PDB, or HPA-starved workloads that cannot place a replacement because the pool is at capacity. Fix the budget or add surge capacity, and the drain resumes; forcing a node delete instead converts a controlled upgrade into an outage.

• A node pool stuck in REPAIRING usually means auto-repair keeps failing the same health check (kubelet not ready after reboot). Read the pool's conditions and the node's events; a bad DaemonSet or a node-local disk error will loop repair forever.
• Node image incompatibility: workloads pinned to kernel features or hostPath state that the new image changes — test in a staging pool first (--node-pool upgrade order matters).
• After a failed node-pool upgrade, check version skew: control plane versus nodes may now sit at the edge of the supported skew window, which blocks the next step until you finish or roll the pool forward.

Upgrades with surge nodes need temporary capacity: CPU quota in the region and secondary IP ranges for new pods. The operation error says which (Quota 'CPUS' exceeded, or no free IP addresses in subnet). Fixes: request quota before the window; widen the pod range or reduce --max-pods-per-node planning; lower --surge/--max-surge so the upgrade fits inside what you have.

1. Let the failed operation finish or be aborted cleanly — never parallel operations.
2. Fix the named blocker (PDB, quota, repair loop).
3. Upgrade node pools explicitly after the control plane, one pool at a time, watching kubectl get nodes and workload readiness between pools.
4. Keep --maintenance-window and release-channel expectations aligned with the team; surprise auto-upgrades are a process bug, not a cluster bug.

Each minor version retires API versions; manifests still using them apply fine today and fail after the control plane moves. The warnings are already visible if you look: kubectl prints deprecation warnings on apply, and audit/scanning tools (kubent, pluto) list every object on a dead version per cluster. Run that scan before scheduling, fix manifests in CI (where the same deprecation warnings should fail the pipeline), and only then upgrade the control plane. The post-upgrade symptom is distinctive: new deploys error with "no matches for kind ... in version" while existing workloads keep running — a manifest problem, not a cluster problem, but it pages like one.

Release channels decide when Google can move your version; a cluster on Rapid that "upgraded itself" mid-quarter is the channel doing its documented job. Keep maintenance windows configured so enforced upgrades land in staffed hours, and read the channel's release notes the way you read dependency changelogs. Node auto-upgrade follows the control plane within a window — if a pool shows a version you did not choose, check the channel and the auto-upgrade flag before suspecting a rogue actor; and keep one unenrolled staging cluster to rehearse each minor version before the production window.

• Audit PDBs before upgrade season; a PDB that can never allow a drain is a landmine with a schedule.
• Reserve surge quota as part of the upgrade runbook, not during the incident.
• Stage upgrades: one test pool, then the rest; read node events between.
• Subscribe to GKE release notes for your channel; know which version removes the API your manifests still use (kubectl explain + deprecation warnings in CI).