Tokenization and Context Windows
What common mistakes do candidates make around Tokenization and Context Windows, and how do you avoid them?
Answers use simple, clear English.
Quick interview answer
Common mistakes: Assuming 1 token ≈ 1 word globally. Best practices: Count tokens with the model tokenizer; truncate/summarize for limits.
Detailed answer
Common mistakes: Assuming 1 token ≈ 1 word globally. Best practices: Count tokens with the model tokenizer; truncate/summarize for limits. Core: BPE/WordPiece/Unigram split text into tokens. Cost and context limits are in tokens not words. Multilingual and code tokenize differently. Real-time example: A 4k-character JSON may be 2k–8k tokens depending on tokenizer. Pros: Open vocabulary; handles rare words via subwords. Cons: Token tax on whitespace/JSON; surprises in cost estimation. Common mistakes: Assuming 1 token ≈ 1 word globally. Best practices: Count tokens with the model tokenizer; truncate/summarize for limits. Audience level: Senior.
Full explanation
BPE/WordPiece/Unigram split text into tokens. Cost and context limits are in tokens not words. Multilingual and code tokenize differently.
Real example & use case
A 4k-character JSON may be 2k–8k tokens depending on tokenizer.
Pros & cons
Pros: Open vocabulary; handles rare words via subwords. Cons: Token tax on whitespace/JSON; surprises in cost estimation.
Common mistakes
Assuming 1 token ≈ 1 word globally.
Best practices
Count tokens with the model tokenizer; truncate/summarize for limits.
Follow-up questions
Open one as its own read / solve / listen card