Tokenization
also: BPE
Splitting text into tokens, usually with byte-pair encoding that merges frequent character pairs.
Tokenization converts raw text into the token IDs a model consumes. Byte-pair encoding (BPE) starts from characters and repeatedly merges the most frequent adjacent pair, growing a vocabulary that balances short sequences against a manageable vocab size. Common words become a single token; rare words, code, and other languages split into several.
Worked example: in English, one token is roughly 4 characters (~0.75 words), so “tokenization” might be one or two tokens while a long API key or a Chinese sentence becomes many. Gotcha: a token is not a word or a character, and everything is priced and budgeted in tokens — the same sentence can cost noticeably more in a language or format the tokenizer handles poorly, and that is invisible until you count.