Kafka Exactly-Once Semantics
Go deep on Kafka Exactly-Once Semantics: edge cases, scalability limits, and how you'd evolve the solution over 2 years.
Answers use simple, clear English.
Quick interview answer
Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern.
Detailed answer
Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern. Scale limits often appear in: Higher latency; transactional overhead; not magic for external DB writes.. Evolution levers: Outbox table + relay; idempotency keys on consumers.. Anchor example: Stream processor reads payments, updates balance topic and audit topic atomically in one transaction. Core: Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern. Real-time example: Stream processor reads payments, updates balance topic and audit topic atomically in one transaction. Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes. Common mistakes: Assuming EOS covers DB update without idempotency key. Best practices: Outbox table + relay; idempotency keys on consumers. Audience level: Senior.
Full explanation
Idempotent producer + transactional writes (read-process-write in one transaction) give EOS within Kafka. End-to-end EOS needs idempotent sinks or outbox pattern.
Real example & use case
Stream processor reads payments, updates balance topic and audit topic atomically in one transaction.
Pros & cons
Pros: No duplicate downstream messages within pipeline. Cons: Higher latency; transactional overhead; not magic for external DB writes.
Common mistakes
Assuming EOS covers DB update without idempotency key.
Best practices
Outbox table + relay; idempotency keys on consumers.
Follow-up questions
Only answered follow-ups are shown — click to open with full answers