Amazon SQS
Durable message queue mechanics: retention, event source mapping to Lambda, batch size vs reserved concurrency, and dead-letter queues.
A managed, durable queue. Absorbs a burst of messages instantly and holds them until something is ready to process them, up to 14 days retention at the maximum setting, 4 days by default.
Why this matters: the producer never has to know or care whether the consumer is keeping up in real time. A queue is the boundary between what you control (how fast you drain it) and what you don't (an external system's real capacity).
Connecting to Lambda: event source mapping
The configuration (not code) that tells Lambda which queue to poll and how to turn messages into invocations. Two separate dials control how aggressively it processes:
- Batch size: how many messages one invocation receives at once. Doesn't cap concurrency on its own, Lambda will still scale out to many parallel single-message invocations if the queue grows.
- Reserved concurrency: the actual throttle. Caps how many invocations run at once, regardless of queue depth. Tune this to what a downstream system can survive, not to what AWS allows.
Dead-letter queues
A second SQS queue attached to the main one via a redrive policy, a rule that watches how many times a message has been received and moves it once it crosses a maximum receive count. Gives a failed message a hard ceiling instead of retrying forever, and a dedicated place with its own consumer to handle it (notify the user, log it, whatever the failure needs).
Where this showed up
- The number of requests isn't the problem: batch size, reserved concurrency, throttling an external system.
- Three tries, then a different path: dead-letter queues and redrive policy.