Idempotent Consumers with Kafka
How do you design an idempotent Kafka consumer so redelivered messages do not double-apply side effects?
Answers use simple, clear English.
Audio N/AQuick interview answer
Store processed event IDs (or partition+offset) in a unique constraint table, or make the business write upsert/idempotent. Use transactional outbox for producing. At-least-once delivery is common — consumers must tolerate duplicates.
Detailed answer
Store processed event IDs (or partition+offset) in a unique constraint table, or make the business write upsert/idempotent. Use transactional outbox for producing. At-least-once delivery is common — consumers must tolerate duplicates.
Real example & use case
Payment settled event credits a wallet once even if the message is redelivered after a crash.
Pros & cons
Pros: safe retries. Cons: need durable idempotency store; exactly-once is nuanced across systems.