Kafka Exactly-Once Semantics
What are the key trade-offs of Kafka Exactly-Once Semantics? When would you choose it vs alternatives?
Answers use simple, clear English.
Audio N/AQuick interview answer
Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes.
Detailed answer
Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes. Decision: pick when benefits outweigh operational cost. Core: Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern. Real-time example: Stream processor reads payments, updates balance topic and audit topic atomically in one transaction. Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes. Common mistakes: Assuming EOS covers DB update without idempotency key. Best practices: Outbox table + relay; idempotency keys on consumers. Audience level: Junior.
Full explanation
Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern.
Real example & use case
Stream processor reads payments, updates balance topic and audit topic atomically in one transaction.
Pros & cons
Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes.
Common mistakes
Assuming EOS covers DB update without idempotency key.
Best practices
Outbox table + relay; idempotency keys on consumers.
Follow-up questions
- How would you test Kafka Exactly-Once Semantics?
- What metrics prove Kafka Exactly-Once Semantics is healthy in prod?
- How does Kafka Exactly-Once Semantics change at 10× traffic?