Consumer Groups and Rebalance
As a Engineering Manager engineer, explain Consumer Groups and Rebalance. What problem does it solve and how would you describe it in an interview?
Answers use simple, clear English.
Audio N/AQuick interview answer
At Engineering Manager depth: start with the problem, then mechanism, then a short example. Each partition consumed by one consumer in group.
Detailed answer
At Engineering Manager depth: start with the problem, then mechanism, then a short example. Each partition consumed by one consumer in group. Rebalance on member join/leave — can cause duplicate processing if offsets not committed carefully. Core: Each partition consumed by one consumer in group. Rebalance on member join/leave — can cause duplicate processing if offsets not committed carefully. Real-time example: 6 partitions, 3 consumers → each gets 2 partitions; add 4th consumer triggers rebalance. Pros: Elastic scale-out; at-least-once default with offset commit. Cons: Rebalance pauses consumption; sticky assign reduces churn. Common mistakes: Long processing without pause; committing before side effects complete. Best practices: Idempotent consumers; static membership; cooperative sticky assignor. Audience level: Engineering Manager.
Full explanation
Each partition consumed by one consumer in group. Rebalance on member join/leave — can cause duplicate processing if offsets not committed carefully.
Real example & use case
6 partitions, 3 consumers → each gets 2 partitions; add 4th consumer triggers rebalance.
Pros & cons
Pros: Elastic scale-out; at-least-once default with offset commit. Cons: Rebalance pauses consumption; sticky assign reduces churn.
Common mistakes
Long processing without pause; committing before side effects complete.
Best practices
Idempotent consumers; static membership; cooperative sticky assignor.
Follow-up questions
- How would you test Consumer Groups and Rebalance?
- What metrics prove Consumer Groups and Rebalance is healthy in prod?
- How does Consumer Groups and Rebalance change at 10× traffic?