Check the Fuse Before You Try the Call
A circuit breaker built from a DynamoDB heartbeat table with a TTL, so a Lambda stops hammering an already-down external system instead of retrying blind.
The dead-letter queue from the last post solves one problem: a registration that keeps failing eventually stops retrying forever and tells the user instead of going silent. It doesn't solve a different problem sitting right next to it.
While the RMV system is down, every single message still gets a full attempt. Each one hits its own retries inside the Lambda function before giving up, gets pushed back to the queue, and the next message in line does the exact same thing. A struggling system gets pounded by exactly the traffic it can least afford while it's already failing.
The fix is a fuse, not more retries
A circuit breaker does what the name says: it breaks the circuit. Once it's clear a downstream system isn't responding, stop sending it requests entirely for a while, instead of learning that lesson fresh on every single message.
The mechanism here doesn't need a dedicated circuit breaker service. It's a DynamoDB table.
ExpandBefore calling the RMV system, the Lambda checks a heartbeat table. A live "down" record short-circuits the call; no record means try again
How the heartbeat table works
Call it a heartbeat table. It holds one thing: whether a given external system is currently known to be down.
- The Lambda tries the RMV call and it fails, after its own internal retries, same as before.
- It writes a record to the heartbeat table: something like
system: RMV, status: down, with a TTL attribute set to expire in 5 minutes. - DynamoDB deletes that record automatically once the TTL passes. No cleanup Lambda, no scheduled job, the database does this on its own.
The short-circuit happens on the read side. Before the Lambda calls RMV, it checks the heartbeat table for a live record first.
- Record exists? Skip the call entirely. Return immediately, the message goes back to the queue to be tried later, same as any other retry, but this time without wasting a call on a system already known to be down.
- No record? Make the real call. Either it succeeds and processing continues normally, or it fails again and writes a fresh heartbeat entry with a new 5-minute TTL.
Why the TTL detail matters
A naive version of this table would need something to clean up stale "down" records once the system recovers, a scheduled job polling the table, deleting old entries. DynamoDB's TTL feature removes that need entirely: the record expires on its own, and once it's gone, the very next attempt tests reality again.
The classic circuit breaker pattern names three states: closed, calls go through normally; open, calls are blocked outright; and half-open, a cautious trial period where exactly one call is allowed through to test whether the problem is actually fixed. This design doesn't build all three as explicit states, it's really just a record that exists or doesn't. But the effect lines up with half-open anyway: once the TTL expires, exactly one path finds out whether the downstream system actually recovered, rather than every queued message hammering it at once to find out.
What this buys, concretely
Two things, and they're related.
- Less load on a system that's already struggling, at precisely the moment it can least afford more.
- An explicit, queryable signal of downstream health. The heartbeat table isn't just internal plumbing, it's a place to check, or build a dashboard against, when someone asks "is the RMV integration currently healthy?"
Building this by hand in a single Lambda function is reasonable for one external dependency. For a system with more than one, or when the pattern needs to combine with retries and exponential backoff, a resiliency library is worth reaching for instead of hand-rolling the same logic repeatedly. For Node.js, Cockatiel provides circuit breaker, retry, and backoff utilities as composable building blocks rather than custom code per integration.
With the failure path optimized on the vehicle side, the last piece of the system design is the one users touch every time they come back: logging in.
The Essentials
- Retrying blind during an outage makes a bad situation worse. Every queued message hammering a downed system delays its recovery instead of waiting for it.
- A DynamoDB record with a TTL is a lightweight circuit breaker, no dedicated service or scheduled cleanup job required.
- Check before you call, not after. The short-circuit happens on the read path, before the Lambda spends a network call finding out what it already knows.
- TTL expiry acts like a half-open state. The first attempt after the cooldown finds out whether the system actually recovered.
- A dedicated resiliency library beats hand-rolled logic per integration. Cockatiel, for Node.js, packages circuit breakers, retries, and backoff as reusable pieces.
Further Reading and Watching
Keep reading