Kafka Exactly-Once Semantics
As a Junior engineer, explain Kafka Exactly-Once Semantics. What problem does it solve and how would you describe it in an interview?
Answers use simple, clear English.
Quick interview answer
At Junior depth: start with the problem, then mechanism, then a short example. Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka.
Detailed answer
At Junior depth: start with the problem, then mechanism, then a short example. Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern. Core: Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern. Real-time example: Stream processor reads payments, updates balance topic and audit topic atomically in one transaction. Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes. Common mistakes: Assuming EOS covers DB update without idempotency key. Best practices: Outbox table + relay; idempotency keys on consumers. Audience level: Junior.
Full explanation
Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern.
Real example & use case
Stream processor reads payments, updates balance topic and audit topic atomically in one transaction.
Pros & cons
Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes.
Common mistakes
Assuming EOS covers DB update without idempotency key.
Best practices
Outbox table + relay; idempotency keys on consumers.
Follow-up questions
- How would you test Kafka Exactly-Once Semantics?
- What metrics prove Kafka Exactly-Once Semantics is healthy in prod?
- How does Kafka Exactly-Once Semantics change at 10× traffic?