Three Tries, Then a Different Path
What happens when the external RMV system won't answer at all: a bounded number of retries, a dead-letter queue, and a Lambda whose only job is telling the user.
Capping concurrency at 5, from the last post, protects the DMT system from a traffic spike it can't handle. It does nothing for the scenario where that system simply isn't answering at all, offline, timing out, or failing every request it receives.
That's a different failure mode, and it needs its own answer.
What happens when a call fails
The rate-limited Lambda from the last post tries to reach the external system and doesn't get a response. It retries a handful of times on its own. If those retries also fail, the message doesn't just vanish, it goes back onto the SQS queue.
That's the normal behavior of an SQS-triggered Lambda: a message that isn't explicitly deleted (which only happens on a successful run) becomes visible again and gets picked up for another attempt.
A queue can't retry forever
Left alone, that message would cycle through the queue indefinitely: picked up, failed, returned, picked up again. SQS has a setting that stops this, called a redrive policy, a rule that watches how many times a message has been picked up and moves it somewhere else once it crosses a maximum receive count.
Set that number to 3, and here's what happens.
- Attempt 1 fails. The message goes back to the queue.
- Attempt 2 fails. Same thing.
- Attempt 3 fails. This time, instead of returning to the main queue, the message gets redirected to a dead-letter queue, a second SQS queue that exists purely to catch messages the main flow couldn't process.
ExpandAfter three failed attempts, a message moves from the main queue to a dead-letter queue, where a separate Lambda notifies the user by SMS instead of retrying forever
The number 3 isn't arbitrary. It has to be high enough that a brief network blip or a momentary timeout on the RMV system's end doesn't immediately give up, but low enough that the system stops hammering an external service that's genuinely down.
What the dead-letter queue is actually for
A dead-letter queue isn't a place where failed messages go to be forgotten. It's a landing zone with its own consumer, and in this design, that consumer is a small Lambda function with exactly one job.
Call it the notify-failure Lambda. It doesn't retry the RMV check, doesn't attempt to fix anything, doesn't touch the original registration data beyond reading it. It sends an SMS telling the user their vehicle registration didn't go through, and that they'll be notified once it's tried again.
That's a deliberately narrow scope. Mixing retry logic and user notification into the same function would mean one function doing two unrelated jobs, and a bug in either one risks breaking the other.
Why this matters more than it looks
The alternative to this design is silence: a registration that quietly fails somewhere in a queue, with no record surfaced to the user and no way for them to know their fuel pass application never completed.
That's the same failure this whole series keeps circling back to, from the very first post: an app that goes down without telling anyone what's happening is indistinguishable, from the user's side, from an app that's simply broken. A bounded number of retries followed by a clear notification turns an invisible failure into a visible, explainable one. The user doesn't get their QR code immediately, but they know exactly why, and they know something is still going to happen.
With both branches covered, the successful path through Step Functions and the failure path through the dead-letter queue, the next post finally answers the question this one deferred: what actually happens inside that Step Functions Express Workflow once a vehicle checks out.
The Essentials
- A failed message shouldn't retry forever. A max receive count on the queue's redrive policy puts a hard ceiling on attempts.
- A dead-letter queue is a landing zone with its own consumer, not a place messages disappear to.
- Give the failure-notification Lambda exactly one job. Mixing retry logic and user notification in the same function risks a bug in one breaking the other.
- A visible failure beats a silent one. Users can tolerate a delay if they're told about it. They can't tolerate wondering whether anything happened at all.
- The retry threshold is a judgment call, not a default. High enough to survive a momentary blip, low enough to stop hammering a system that's actually down.
Further Reading and Watching
Keep reading