Circuit Breakers: How to Stop One Failing Service From Taking Down Everything

When a dependency goes down, retries make it worse. Learn how circuit breakers help small systems fail fast, recover gracefully, and stay online.

Your payment gateway starts timing out. Every checkout request now waits thirty seconds before failing. Threads pile up, connection pools drain, and suddenly your whole app is down — not because of a bug in your code, but because you kept politely waiting for a dependency that was never going to answer. This is the classic cascading failure, and the fix has been known for decades: the circuit breaker.

The problem: failure spreads through waiting

Most outages in service-to-service systems aren't caused by errors. They're caused by slowness. An error returns instantly and your code moves on. A hang holds a thread, a socket, and a slot in your connection pool hostage.

When a downstream service degrades, every caller that keeps sending traffic makes two things worse:

• The victim gets hammered by retries exactly when it has the least capacity to recover.
• The caller exhausts its own resources waiting, and becomes the next thing to fall over.

Timeouts help — you should always have them — but a timeout still means every request pays the full waiting cost before failing. When the dependency is clearly down, paying that cost thousands of times is pure waste.

How a circuit breaker works

A circuit breaker wraps calls to a dependency and tracks their outcomes. It has three states:

• Closed — normal operation. Requests flow through, failures are counted.
• Open — too many recent failures. Requests fail immediately without touching the dependency. No waiting, no wasted threads.
• Half-open — after a cooldown, a few trial requests are allowed through. If they succeed, the breaker closes. If they fail, it snaps open again.

The key insight: when a dependency is down, the fastest, kindest thing you can do is stop calling it.

Failing in a millisecond instead of thirty seconds keeps your own service healthy, and the sudden drop in traffic gives the struggling dependency breathing room to recover.

Tuning without overthinking

You don't need fancy math to get value. Reasonable starting points:

• Trip condition: open the circuit if, say, 50% of the last 20 requests failed, or after 5 consecutive failures.
• Cooldown: stay open for 15–30 seconds before probing.
• Count timeouts as failures. A slow success is often the early warning sign, so consider tracking latency too.

One breaker per dependency, not one global breaker — your database and your WhatsApp API endpoint fail independently and should be isolated independently.

What to do when the circuit is open

Failing fast is only half the job. Decide, per feature, what "degraded" looks like:

• Queue it. If a notification can't be sent now, persist it and let a background worker retry later.
• Serve stale data. A cached exchange rate or product list from five minutes ago beats an error page.
• Fail visibly. For payments, tell the user honestly that checkout is temporarily unavailable — a fast, clear error is far better than a spinner that dies after thirty seconds.

Log every state transition. A breaker opening is one of the most useful alerts you can have: it tells you which dependency broke before your users tell you the whole site is down.

The takeaway

Circuit breakers aren't a big-company luxury; most languages have a small library for them, and even a hand-rolled version is under a hundred lines. Combined with sane timeouts and retries with backoff, they turn a dependency outage from a site-wide crisis into a contained, observable, recoverable event. Protect your system from the failures you can't prevent — because you can't prevent most of them.