Graceful Shutdown: The Deploy Bug You Only See in Production

Why processes that die mid-request cause dropped orders and duplicate work — and how to drain connections cleanly before every restart.

Every deploy kills a process. Every autoscaler scale-down kills a process. If your service dies the instant it receives a stop signal, some user somewhere was mid-request — and their order, payment, or message just vanished into a connection reset. The frustrating part is that you'll never see this in development, because nobody deploys while they're clicking around localhost.

Graceful shutdown is the discipline of finishing what you started before you exit. It's a small amount of code, and it eliminates an entire category of "works fine, but sometimes things disappear during deploys" bugs.

What actually happens when a process dies

When your orchestrator, systemd unit, or process manager wants a process gone, it usually sends SIGTERM first, waits a grace period (often 10–30 seconds), then sends SIGKILL. SIGKILL cannot be caught. Everything in flight at that moment — open HTTP responses, half-written files, uncommitted transactions — is simply gone.

A service that ignores SIGTERM gets the worst of both worlds: it keeps accepting new requests during the grace period, then gets killed anyway with even more work in flight.

The correct shutdown sequence

The pattern is the same in Node, Go, Python, or PHP workers:

• Catch SIGTERM and flip an internal "shutting down" flag.
• Stop accepting new work immediately — close the listener, stop pulling from the queue, fail the readiness check so the load balancer routes traffic elsewhere.
• Drain in-flight requests — let current handlers finish, with a hard deadline slightly shorter than the platform's grace period.
• Release resources in order — flush logs and metrics, commit or roll back transactions, close database pools and message broker connections.
• Exit cleanly with code 0 so supervisors don't treat the shutdown as a crash.

The ordering matters. If you close the database pool while requests are still draining, you've just converted a graceful shutdown into a burst of 500 errors.

Readiness checks are half the story

Load balancers and orchestrators only stop sending traffic when the readiness probe fails. If your app closes its listener before the balancer notices, users hit connection refused for a few seconds. The fix is a short delay: fail readiness first, keep serving for a few seconds while the balancer catches up, then close the listener. It feels backwards, but it's the difference between a silent deploy and a spike of errors on every release.

Queue workers need it even more

For background workers, dying mid-job means either a lost job or a duplicate one, depending on when you acknowledge the message. A graceful worker stops fetching new jobs on SIGTERM, finishes the current one, acknowledges it, then exits. Pair this with idempotent job handlers and your deploys become genuinely boring — which is exactly what you want.

A deploy should be invisible to users. If your error dashboard has a small spike at every release, you don't have a reliability problem — you have a shutdown problem.

The takeaway

Test it deliberately: send your service SIGTERM under load and watch what happens. If any request fails, wire up the drain sequence — flag, stop intake, finish work, release resources, exit. It's an afternoon of work, and it turns every future deploy, restart, and scale-down from a small outage into a non-event.