AWS Lambda in production: cold starts, idempotency, and event queues

Executive summary

Lambda removes a lot of infrastructure burden, but it does not remove distributed-systems decisions. In production, recurring incidents usually cluster around three areas:

1. inconsistent latency from initialization;
2. event reprocessing without idempotency;
3. unclear retry and failure-destination policies.

AWS documentation provides the control levers. The key is to treat them as architecture decisions, not late-stage tuning.

1) Cold starts: real issue, contextual solution

AWS recommends execution environment reuse (SDK clients and connections outside the handler) and memory tuning with measurements. For interactive workloads, provisioned concurrency improves startup predictability by keeping execution environments initialized.

Important trade-off:

• provisioned concurrency improves latency consistency;
• it also introduces fixed cost.

Practical decision: apply it where p95/p99 latency is a product requirement, not globally by default.

2) Async does not mean “fire-and-forget without contract”

For asynchronous invocation, Lambda's API docs define default retry behavior (two retries after failures) plus temporary internal queue retention. Without explicit failure destination strategy, events can be dropped after policy exhaustion.

Critical controls:

• define failure destinations (DLQ or on-failure destinations);
• monitor final failure rate, not only first-attempt errors;
• separate transient-error alarms from terminal-failure alarms.

3) Event source mappings: duplication starts here

For Kinesis, DynamoDB Streams, Kafka, and related sources, AWS docs emphasize at-least-once processing behavior. Duplicate deliveries are possible.

Direct implication:

• idempotency is a functional requirement, not an optimization.

Features like BisectBatchOnFunctionError, retry limits, and max record age help isolate poison records, but they do not replace business-level idempotency design.

4) Idempotency must be explicit in domain design

AWS best-practice guidance explicitly recommends idempotent handlers, and Powertools provides implementation utilities.

Minimum pattern:

• business idempotency key (not just raw payload hash);
• key-state persistence with TTL;
• clear semantics for concurrent retries.

Without this, backlog plus retries can amplify side effects (duplicate billing, duplicate notifications, inconsistent updates).

5) Lambda production checklist

1. Define latency budget for critical functions.
2. Choose concurrency strategy by workload class (on-demand, reserved, provisioned).
3. Configure async failure destinations for critical flows.
4. Enforce idempotency on queue/stream-triggered handlers.
5. Instrument replay rate, event age, and discard metrics.

Teams applying this checklist reduce incidents without unnecessary complexity.

Conclusion

Lambda in production is less about "serverless magic" and more about execution contracts: latency, retries, and consistency.

Practical closing question: if the same event is delivered three times in ten minutes, does your system end in the exact same state?

Sources

• Best practices for working with AWS Lambda functions - official documentation
• Understanding Lambda function scaling - official documentation
• Configuring provisioned concurrency for a function - official documentation
• PutFunctionEventInvokeConfig (AWS Lambda API) - official documentation
• Using Lambda to process records from Amazon Kinesis Data Streams - official documentation
• CreateEventSourceMapping (AWS Lambda API) - official documentation
• Idempotency utility (AWS Lambda Powertools) - official documentation