Consumer Groups and Rebalance
As a Engineering Manager engineer, explain Consumer Groups and Rebalance. What problem does it solve and how would you describe it in an interview?
Answers use simple, clear English.
Quick interview answer
At Engineering Manager depth: start with the problem, then mechanism, then a short example. Each partition consumed by one consumer in group.
Detailed answer
At Engineering Manager depth: start with the problem, then mechanism, then a short example. Each partition consumed by one consumer in group. Rebalance on member join/leave — can cause duplicate processing if offsets not committed carefully. Core: Each partition consumed by one consumer in group. Rebalance on member join/leave — can cause duplicate processing if offsets not committed carefully. Real-time example: 6 partitions, 3 consumers → each gets 2 partitions; add 4th consumer triggers rebalance. Pros: Elastic scale-out; at-least-once default with offset commit. Cons: Rebalance pauses consumption; sticky assign reduces churn. Common mistakes: Long processing without pause; committing before side effects complete. Best practices: Idempotent consumers; static membership; cooperative sticky assignor. Audience level: Engineering Manager.
Full explanation
Each partition consumed by one consumer in group. Rebalance on member join/leave — can cause duplicate processing if offsets not committed carefully.
Real example & use case
6 partitions, 3 consumers → each gets 2 partitions; add 4th consumer triggers rebalance.
Pros & cons
Pros: Elastic scale-out; at-least-once default with offset commit. Cons: Rebalance pauses consumption; sticky assign reduces churn.
Common mistakes
Long processing without pause; committing before side effects complete.
Best practices
Idempotent consumers; static membership; cooperative sticky assignor.
Follow-up questions
- How would you test Consumer Groups and Rebalance?
- What metrics prove Consumer Groups and Rebalance is healthy in prod?
- How does Consumer Groups and Rebalance change at 10× traffic?