Health Checks That Tell the Truth: Liveness, Readiness, and Lying Endpoints

Most /health endpoints return 200 while the app is broken. How to design liveness and readiness checks that reflect what users actually experience.

Almost every service has a /health endpoint, and most of them lie. They return 200 OK because the process is up and the web framework can serve a route — while the database connection pool is exhausted, the message queue is unreachable, and every real request is failing. The load balancer keeps sending traffic to a server that can answer exactly one question: "are you running?" That was never the question that mattered.

Two questions, two endpoints

The confusion usually starts by cramming two different questions into one check.

• Liveness asks: is this process beyond saving? If yes, restart it. A liveness check should be dumb and cheap — return 200 if the process can execute code. Deadlocked event loop, corrupted internal state, out-of-memory death spiral: those are liveness failures.
• Readiness asks: should this instance receive traffic right now? This is where dependencies belong. If the app can't reach its database, it isn't ready — but restarting it won't fix the database, so it shouldn't fail liveness.

Mixing them up has a classic failure mode: you put a database ping in the liveness check, the database has a 30-second blip, and your orchestrator restarts every healthy instance at once. A dependency outage becomes a full-fleet restart storm. Readiness removes an instance from rotation; liveness kills it. Choose accordingly.

Check what a request actually needs

A good readiness check exercises the same path a real request takes, in miniature:

• Database: run SELECT 1 on a connection from the pool, not a fresh one. A fresh connection can succeed while the pool is fully leaked.
• Cache and queue: a lightweight ping is fine, but decide whether each dependency is hard or soft. If your app degrades gracefully without Redis, don't fail readiness over it — you'd be turning a partial outage into a total one.
• Disk and config: if the service writes files or loaded a critical config at boot, verify it once and cache the result briefly.

A health check should predict whether the next real request will succeed. Anything it verifies beyond that is noise; anything less is a lie.

Keep the whole check under a few hundred milliseconds and cache results for a second or two. Load balancers poll frequently, and you don't want health traffic itself to strain the database.

Fail on the way up, and on the way down

The two moments health checks earn their keep are startup and shutdown.

On startup, stay unready until migrations have run, caches are warmed, and connections are established. Flipping ready too early means your fresh deploy serves errors for its first few seconds — which looks exactly like a bad release.

On shutdown, do the reverse: when the process receives SIGTERM, immediately start failing readiness, keep serving in-flight requests, wait for the balancer to notice, then exit. This one habit eliminates most of the mysterious 502s that appear only during deploys.

Alert on symptoms, page on checks carefully

Health endpoints are for machines making routing decisions, not a substitute for observability. A service can pass every check while returning garbage data. Pair checks with real signals — error rates, latency percentiles, queue depth — and treat a failing readiness probe as a routing event first and an alert second.

The takeaway: split liveness from readiness, test the real request path with pooled resources, distinguish hard dependencies from soft ones, and wire shutdown to drain gracefully. A health check that tells the truth turns outages from mysteries into non-events — the traffic just quietly goes somewhere healthy.