Byte-Pair Encoding
also: BPE
The algorithm most tokenizers use: repeatedly merge the most frequent adjacent symbol pair to build a subword vocabulary.
Byte-pair encoding (BPE) builds a tokenizer’s vocabulary by starting from individual characters or bytes and repeatedly merging the most frequent adjacent pair into a new token, until it reaches the target vocabulary size. The result is subword units — common words become one token, rare words split into pieces.
Worked example: “tokenization” might split into “token” + “ization” while “the” is a single token; a novel string like a long hash falls back to many small pieces. Gotcha: because it is frequency-driven on the training corpus, BPE tokenizes languages and code unevenly — text in under-represented languages costs more tokens, a real cost and context-length disadvantage.