Cloud SQL Won't Connect: Picking the Right Path and Debugging It
Cloud SQL connections fail in the space between your client and the instance: the access path you chose, the identity you present, and the database user you then become. Separate those three and every error has a home.
First: which path are you on?
- Public IP + authorized networks — the client's egress IP must be listed; a NAT change breaks it.
- Private IP (VPC peering / PSC) — the client must be inside the peered VPC; from anywhere else the address simply does not route.
- Cloud SQL Auth Proxy / connector — the proxy authenticates with IAM and tunnels; failures are proxy-side or IAM-side.
A "worked yesterday" failure on public IP is an egress-IP change nine times out of ten. A private-IP timeout from a laptop is not an incident, it is routing.
Proxy and connector failures
./cloud-sql-proxy project:region:instance --port 5433 # watch its log line: "Authorized to connect" vs the error
| Proxy/connector error | Cause |
|---|---|
403 ... does not have permission / cloudsql.instances.connect missing | The IAM principal lacks the connect role; service accounts in Cloud Run/Functions need it explicitly |
404 instance not found | Wrong connection name format (project:region:instance) or wrong project |
Proxy connects but the app gets password authentication failed | Proxy handles the network auth; the DB user/password (or IAM DB auth) is still required and is wrong |
IAM database authentication pitfalls
With IAM auth (Postgres/MySQL), the token is the password and tokens expire in an hour. Classic breakage: an app caches the token, works at deploy, fails 61 minutes later. Use the connector libraries that refresh tokens, not a hand-rolled gcloud auth print-access-token baked into an env var. Also verify the IAM user was actually created inside the database (CREATE USER for the IAM principal) — enabling IAM auth on the instance does not create logins.
Instance-side refusals
gcloud sql instances describe INSTANCE --format="value(state,ipAddresses)" gcloud sql operations list --instance=INSTANCE --limit=5
state: MAINTENANCEorPENDING_*— brief windows where connections drop; schedule around them.- Connection count exhausted: too-small tier plus a pool that never shrinks; the operations log shows "too many connections".
- Storage autoresize hit its cap, or the instance is out of disk and refuses writes — check
databaseDiskQuotausage.
Private Service Connect and peering gotchas
Private IP setup creates a VPC peering (servicenetworking) connection. If a later project change deletes or conflicts with the peering range, new connections fail with obscure routing behaviour while existing flows limp along. Verify the peering exists and the allocated range is not reused: gcloud compute networks peerings list --network=NET and gcloud compute addresses list --purpose=VPC_PEERING.
Connection-pool exhaustion: the slow denial
Every Cloud SQL tier has a hard connection ceiling; the failure mode is not a firewall error but refusals like remaining connection slots are reserved (Postgres) or Too many connections (MySQL). The arithmetic that bites: 20 pods × 10 pool size × 2 environments sharing one instance exceeds the tier before anyone notices. Watch pg_stat_activity counts or SHOW STATUS LIKE 'Threads_connected' against the tier limit, put a pooler (PgBouncer, or the connector's built-in pooling) in front of chatty fleets, and hunt the leak pattern — pools that are created per-request or never closed after deploys show as connection counts that climb stepwise with each rollout. Right-sizing the pool is usually a two-line change; the incident is the discovery cost.
Client error → layer, at a glance
| Error | Layer | First check |
|---|---|---|
| timeout to private IP | routing | is the client inside the peered VPC? |
| timeout to public IP | authorized networks | current egress IP vs list |
| 403 in proxy log | IAM | cloudsql.client + connect roles |
| password/IAM auth failed | database login | user exists in DB; token age |
| too many connections | capacity | pool math vs tier limit |
Failover and maintenance: connections that drop by design
During a failover or maintenance window every existing connection is dropped; an application without reconnect-with-retry turns a 30-second event into a paging incident. Verify your driver/pooler retries on connection (not just on query), and test the behaviour deliberately: gcloud sql instances failover INSTANCE in staging, watch the app log, and confirm it recovers unattended. A cheap end-to-end canary for the whole path — network, IAM, DB login — is one SELECT 1 through the same connector your app uses, run every few minutes from the real runtime, with the result fed to your alerting.
Prevention checklist
- Standardise on the connector/proxy path; fewer moving parts than authorized networks.
- Grant
cloudsql.client+cloudsql.instances.connectat provisioning time and document the DB-side IAM user creation. - Alert on instance state changes and on connection-count percentage of the tier limit.
- Test connections from the real runtime (Cloud Run VPC connector, GKE node pool), not from a laptop, during setup.