Is it possible to do this?
When dealing with SQS event streams, if the Lambda function does not have adequate reserved concurrency, the function will be throttled, and the unprocessed events / messages can be retried via the SQS redrive policy. I've never liked this limitation as unprocessed messages will eventually end up on the DLQ after some arbitrary number of retries / message visibility timeout.
From my naive perspective, it would appear that the above solution would not be possible with MSK, as placing a message back on an MSK topic for some visibility timeout would effectively lose topic delivery order.
I've searched around but can't find any detail as to how back pressure can be implemented with MSK to Lambda. Does anybody have any insight into how the MSK topic consumer handles Lambda throttling?
Many thanks!
question from:https://stackoverflow.com/questions/65644736/how-to-apply-back-pressure-with-aws-lambda-and-msk-kafka-event-stream