Observability: Metrics Logs Traces
You are a Mid-level on-call. A production issue might involve Observability: Metrics Logs Traces. How do you diagnose and mitigate?
Answers use simple, clear English.
Audio N/AQuick interview answer
Mitigate first, then root-cause. Check symptoms against: Paging on raw CPU; no runbooks..
Detailed answer
Mitigate first, then root-cause. Check symptoms against: Paging on raw CPU; no runbooks.. Validate with: SLO-based alerts; sampling; exemplars linking metrics→traces.. Context: Metrics for SLIs/SLOs, structured logs, distributed traces. Correlate with TraceId. Alert on symptoms (latency/errors) not only CPU. Core: Metrics for SLIs/SLOs, structured logs, distributed traces. Correlate with TraceId. Alert on symptoms (latency/errors) not only CPU. Real-time example: p99 latency burn alert → trace shows slow DB span → log shows lock wait. Pros: Faster incident diagnosis. Cons: Cost/cardinality explosions. Common mistakes: Paging on raw CPU; no runbooks. Best practices: SLO-based alerts; sampling; exemplars linking metrics→traces. Audience level: Mid-level.
Full explanation
Metrics for SLIs/SLOs, structured logs, distributed traces. Correlate with TraceId. Alert on symptoms (latency/errors) not only CPU.
Real example & use case
p99 latency burn alert → trace shows slow DB span → log shows lock wait.
Pros & cons
Pros: Faster incident diagnosis. Cons: Cost/cardinality explosions.
Common mistakes
Paging on raw CPU; no runbooks.
Best practices
SLO-based alerts; sampling; exemplars linking metrics→traces.
Follow-up questions
- How would you test Observability: Metrics Logs Traces?
- What metrics prove Observability: Metrics Logs Traces is healthy in prod?
- How does Observability: Metrics Logs Traces change at 10× traffic?