Tokenization
also: BPE · byte-pair encoding
Splitting text into tokens, usually with byte-pair encoding that merges frequent character pairs.
Tokenization converts raw text into the token IDs a model consumes. Byte-pair encoding (BPE) starts from characters and repeatedly merges the most frequent adjacent pair, growing a vocabulary that balances short sequences against a manageable vocab size.