Transformer Self-Attention
What are the key trade-offs of Transformer Self-Attention? When would you choose it vs alternatives?
Answers use simple, clear English.
Audio N/AQuick interview answer
Pros: Parallel training; long-range dependencies better than RNNs. Cons: O(n²) memory/time in sequence length.
Detailed answer
Pros: Parallel training; long-range dependencies better than RNNs. Cons: O(n²) memory/time in sequence length. Decision: pick when benefits outweigh operational cost. Core: Queries/Keys/Values compute attention weights; each token aggregates context. Multi-head captures different relations. Positional encodings add order. Real-time example: In 'The animal didn't cross the street because it was tired', attention links 'it'→'animal'. Pros: Parallel training; long-range dependencies better than RNNs. Cons: O(n²) memory/time in sequence length. Common mistakes: Confusing encoder-only (BERT) vs decoder-only (GPT) vs encoder-decoder (T5). Best practices: Know masking in causal LMs; KV-cache for inference speed. Audience level: Architect.
Full explanation
Queries/Keys/Values compute attention weights; each token aggregates context. Multi-head captures different relations. Positional encodings add order.
Real example & use case
In 'The animal didn't cross the street because it was tired', attention links 'it'→'animal'.
Pros & cons
Pros: Parallel training; long-range dependencies better than RNNs. Cons: O(n²) memory/time in sequence length.
Common mistakes
Confusing encoder-only (BERT) vs decoder-only (GPT) vs encoder-decoder (T5).
Best practices
Know masking in causal LMs; KV-cache for inference speed.
Follow-up questions
- How would you test Transformer Self-Attention?
- What metrics prove Transformer Self-Attention is healthy in prod?
- How does Transformer Self-Attention change at 10× traffic?