Transformer Self-Attention
What common mistakes do candidates make around Transformer Self-Attention, and how do you avoid them?
Answers use simple, clear English.
Audio N/AQuick interview answer
Common mistakes: Confusing encoder-only (BERT) vs decoder-only (GPT) vs encoder-decoder (T5). Best practices: Know masking in causal LMs; KV-cache for inference speed.
Detailed answer
Common mistakes: Confusing encoder-only (BERT) vs decoder-only (GPT) vs encoder-decoder (T5). Best practices: Know masking in causal LMs; KV-cache for inference speed. Core: Queries/Keys/Values compute attention weights; each token aggregates context. Multi-head captures different relations. Positional encodings add order. Real-time example: In 'The animal didn't cross the street because it was tired', attention links 'it'→'animal'. Pros: Parallel training; long-range dependencies better than RNNs. Cons: O(n²) memory/time in sequence length. Common mistakes: Confusing encoder-only (BERT) vs decoder-only (GPT) vs encoder-decoder (T5). Best practices: Know masking in causal LMs; KV-cache for inference speed. Audience level: Tech Lead.
Full explanation
Queries/Keys/Values compute attention weights; each token aggregates context. Multi-head captures different relations. Positional encodings add order.
Real example & use case
In 'The animal didn't cross the street because it was tired', attention links 'it'→'animal'.
Pros & cons
Pros: Parallel training; long-range dependencies better than RNNs. Cons: O(n²) memory/time in sequence length.
Common mistakes
Confusing encoder-only (BERT) vs decoder-only (GPT) vs encoder-decoder (T5).
Best practices
Know masking in causal LMs; KV-cache for inference speed.
Follow-up questions
- How would you test Transformer Self-Attention?
- What metrics prove Transformer Self-Attention is healthy in prod?
- How does Transformer Self-Attention change at 10× traffic?