NGINX 502 and 504: Reading Upstream Errors Like the Error Log Does

Everything starts in the error log, because the access log only says 502. The upstream error string is the diagnosis:

tail -f /var/log/nginx/error.log
# example:
2026-08-29 10:14:03 [error] 1234#0: *5678 connect() failed (111: Connection refused)
  while connecting to upstream, client: 203.0.113.9, server: app.example.com,
  request: "GET /api/orders HTTP/1.1", upstream: "http://127.0.0.1:8000/api/orders"

The backend process is down, crashed, or listening on a different port/interface than the proxy_pass target. Check systemctl status for the app, and ss -tlnp | grep :8000 for the listener. After a deploy, this is the classic "the service restarted on a new port" or "the app bound to 127.0.0.1 but NGINX reaches the container IP" mismatch.

The app accepted then closed early: it crashed mid-request, its accept queue overflowed, or a keepalive connection NGINX was reusing was already closed server-side. If it correlates with traffic spikes, raise the app's backlog (e.g. somaxconn, the app server's listen backlog) and enable upstream keepalive with the right order of directives:

upstream app {
    server 127.0.0.1:8000;
    keepalive 32;          # must come AFTER the server lines
}
server {
    location / {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://app;
    }
}

SYN sent, no answer: a firewall drop, a security group, or the upstream IP unreachable from this worker. curl -v telnet://upstream:port from the NGINX host isolates it in seconds.

The connection succeeded but the backend took longer than proxy_read_timeout (default 60s) to answer. The backend is slow, not dead: a long query, a synchronous report, a locked table. Either make the endpoint faster or raise the timeout — but raising it without fixing the slow endpoint just moves the failure to the browser.

proxy_pass http://unix:/run/app.sock: targets a socket the app no longer creates, usually after the app switched to a TCP port or the socket path changed in a deploy. Compare the path in proxy_pass with ls -l /run/ and the app's config.

A 502 with upstream sent too big header or upstream prematurely closed connection while reading upstream on large responses is a buffering problem: the response header or body exceeds NGINX's buffers and the temp directory cannot absorb it.

proxy_buffer_size        16k;   # header buffer
proxy_buffers            8 32k;
proxy_busy_buffers_size  64k;
# and make sure the temp path is writable and not full:
df -h /var/lib/nginx

A full disk or a /var/lib/nginx owned by the wrong user produces 502s that look random but follow request size.

Most "NGINX broke after a change" incidents are untested reloads. The discipline is two commands:

nginx -t                 # syntax + file existence
nginx -s reload          # graceful; existing connections finish
# if a reload made things worse:
nginx -s reload          # again, after fixing — reload is not restart
systemctl status nginx   # worker count, recent failures

nginx -T dumps the full effective configuration with includes resolved — the fastest way to find which file really sets the timeout you think you changed.

Every deploy restarts the app; for a second nothing listens on the port and NGINX logs a burst of (111: Connection refused). A few 502s per deploy is normal; hundreds means the app boots slowly and the process manager marks it ready too early. Fix at the app level (readiness = actually serving), and add proxy_next_upstream error http_502; with a second backend so a refused connection retries the healthy node instead of surfacing.

When 504s appear across unrelated endpoints at once, the backend is not slow — it is blocked, usually on a database lock. The NGINX log shows upstream timed out, the app log shows nothing (requests never finish), and the database shows an idle-in-transaction blocker. Chasing NGINX timeouts wastes the incident; the queue of waiting upstream connections is the clue to look downstream.

proxy_pass speaks HTTP/1.1 to the upstream by default. A gRPC service behind NGINX needs grpc_pass with an http2 upstream, or every call fails in ways that look like random 502s. If the backend is gRPC, the location block must use the grpc directives, not the proxy ones.

• Add upstream health checks (active via health_check on NGINX Plus, or passive max_fails/fail_timeout) so one dead backend stops taking traffic.
• Log $upstream_response_time and $upstream_status in the access log — you then see backend latency and backend status without opening the error log.
• Alert on 5xx rate per upstream, not per site; a single misbehaving backend hides in a site-wide number.
• Keep proxy_connect_timeout low and retries (proxy_next_upstream) limited to idempotent methods.