Why don't LLMs just use words as tokens?

A pure word vocabulary would be huge, would miss every typo, new word, or rare name (all become 'unknown'), and would waste capacity on inflections like run/runs/running. Character tokens avoid that but make sequences very long and force the model to relearn spelling of every word. Sub-word tokenization (BPE) is the sweet spot: common words are one token, rare ones split into reusable pieces, and nothing is ever truly unknown.

Why does token count matter for cost and context?

LLMs are billed per token and have a fixed context window measured in tokens, not words or characters. Fewer tokens for the same text means lower cost and more room in the window. Because tokenizers pack common English efficiently but split rare words, code, numbers, and many non-English languages into more tokens, the exact same idea can cost very different amounts depending on how it is written.

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LABS · 02 / TOKENIZATION

The Tokenizer.

Q: What is tokenization in an LLM?

Tokenization is the step that turns raw text into the units a language model actually reads — tokens. A token is usually a sub-word chunk (like 'token' + 'ization'), not a whole word or a single character. Each token is mapped to an integer ID, and the model only ever sees those IDs. Everything about context limits, cost, and even some model 'mistakes' traces back to how text was tokenized.

Q: What is Byte-Pair Encoding (BPE)?

Byte-Pair Encoding is the algorithm behind most LLM tokenizers. It starts with text as individual characters, then repeatedly finds the most frequent adjacent pair of symbols and merges it into a new single token, adding that merge to the vocabulary. After thousands of merges, common words become one token while rare words stay split into a few sub-word pieces — a balance between a tiny character vocabulary and an enormous word vocabulary.

Q: Why do LLMs miscount letters, like the r's in 'strawberry'?

The model never sees individual letters — it sees tokens. 'strawberry' might be a single token or a couple of sub-word chunks, so the model has no direct view of the characters inside them. Counting the r's requires reasoning about the spelling hidden inside tokens, which is exactly the kind of task tokenization makes awkward. It is a tokenization artifact, not a reasoning failure.

A language model can't read letters or words — it reads tokens. Before a single neuron fires, your text is shattered into sub-word chunks and turned into numbers. You'll watch it happen, race three ways to do it, then build the real algorithm by hand.

THE GIST · 20 SECONDS

An LLM only understands token IDs — integers. A tokenizer maps text to those IDs and back. Modern tokenizers use sub-word units: common words become a single token, rare words split into reusable pieces, so nothing is ever truly unknown. The algorithm that finds those pieces is Byte-Pair Encoding — merge the most frequent pair, over and over.

Tokena sub-word chunk — the unit a model reads
Vocabularythe fixed set of tokens the model knows
BPEmerge the most common pair, repeatedly
Token IDthe integer each token maps to

① Watch text become tokens → ② Char vs word vs sub-word → ③ Build BPE by merging pairs → ④ Tune the vocabulary size → ⑤ Cost, context & the 'strawberry' bug

one token just merged word boundary ·

Each colored chip is one token — the unit the model reads. The number under it is its token ID. Notice tokens don't line up with words.

UNDER THE HOOD

What you just played, written down

You shattered text, raced three tokenizers, grew a vocabulary by merging pairs, and felt the cost. Here's the same thing as the algorithm inside every LLM tokenizer.

Byte-Pair Encoding — four steps

Start from characters. Split every word into single characters plus an end-of-word marker. The vocabulary is just the alphabet.
Count pairs. Across the whole corpus, count every adjacent pair of symbols. Find the most frequent one.
Merge it. Replace that pair everywhere with a new single token, and add the merge to the vocabulary. Every sequence gets a little shorter.
Repeat for a fixed number of merges (e.g. ~50,000). Common words collapse to one token; rare words stay split into reusable pieces.

WHY SUB-WORDS WIN

Characters make sequences too long; whole words make the vocabulary huge and choke on anything new. BPE lands in between — frequent = one token, rare = a few pieces — so it's compact and never hits a word it can't represent.

Encoding a word, once trained

text  → "lowest"
chars → l o w e s t </w>      # start split

apply learned merges, in order:
  (l,o)   → lo w e s t         # merge #1
  (lo,w)  → low e s t          # merge #2
  (e,s)   → low es t           # ...
  (es,t)  → low est
tokens → [ low ][ est ]        # 2 tokens, not 6

TINY VOCABlong seqschar-level · model relearns spelling

HUGE VOCABrare tokensword-level · unknowns & sparsity

Solved by hand — BPE on a tiny corpus

Corpus: low · low · lower · newest · widest. Each row is one merge: the most frequent pair, and the new token it becomes. Watch the vocabulary grow and the token count fall.

⚠ Where it bites

Models miscount letters (the r's in strawberry) because they see tokens, not characters. Numbers, code, and rare names split into many tokens — more cost, less context.

↔ Its cousins

WordPiece (BERT) and Unigram (SentencePiece) are alternative sub-word schemes. Modern GPT-style models use byte-level BPE, which never hits an unknown byte.

★ Where you've met it

Every prompt you've ever sent. Token counts drive your bill, your context limit, and even multilingual fairness — some languages cost 2–3× the tokens for the same sentence.

STEP BY STEP

Tokenization, step by step

Byte-pair encoding trained by hand on the corpus from Act 3 (low, lower, newest, widest, wider), how a trained tokenizer encodes new text, and what the vocabulary size costs.

Training and encoding

Count words. Tally each word's frequency, split into characters plus an end marker, and start the vocabulary at those symbols.
Merge the top pair. Count adjacent pairs weighted by word frequency, merge the most common everywhere, and record the merge. Ties go to whichever pair was seen first, which is how this page breaks them.
Stop at a target size. Vocabulary size is the base symbols plus the number of merges. Production tokenizers run tens of thousands of merges.
Encode new text. Split into characters, then replay the merges in the order they were learned. An unseen word falls back to smaller pieces, never to an unknown token.

# one training step
pair = most_frequent_pair(words)
words = merge(words, pair)
vocab.add(pair[0] + pair[1])

Worked example: five merges

Start. Frequencies low 5, lower 2, newest 6, widest 3, wider 2. With end markers that is 107 tokens and 11 symbols.
Merge 1. e+s occurs 9 times (6 in newest, 3 in widest), tied with s+t and t+end; the first seen wins. 98 tokens, vocabulary 12.
Merges 2 and 3. es+t, then est+end, each 9 times. Now 80 tokens, vocabulary 14, and "est" is one token.
Merges 4 and 5. l+o, then lo+w, 7 times each (low 5 plus lower 2). 66 tokens, vocabulary 16, and "low" is one token.
Next. n+e, ne+w and new+est</w> follow at 6 each, reaching 48 tokens at vocabulary 19. "newest" becomes a single token and "widest" splits as w, i, d, est.
Encode "lowest". Its characters merge to low and est, so two tokens plus the end marker, though "lowest" was never in the corpus.

⚠ Vocabulary size is a trade

The embedding table has vocab × d_model entries: 100,000 × 4,096 is 409.6 million parameters, while GPT-2's 50,257 × 768 is 38.6 million. A larger vocabulary shortens sequences, which saves compute and context, but rare tokens get little training.

↔ Bytes and whitespace

Byte-level BPE starts from the 256 byte values, so any string encodes. Most tokenizers fold the leading space into the token, so "the" and " the" are different tokens, and long digit strings are often split into short groups.

★ In production

Count tokens with the exact tokenizer your model uses, since the same text can produce noticeably different counts between families, and non-English text usually costs more. Keep special tokens out of user text, and never mix tokenizers between training and serving.

RUN IT YOURSELF

Train BPE yourself

The lab above, in code: real Byte-Pair Encoding training on a small corpus — CPython in your browser. It repeatedly merges the most frequent pair. Change the number of merges and watch the vocabulary grow. Hit Run (or ⌘/Ctrl + Enter).

HOW TO READ THE CODE — 4 IDEAS

Each word starts as a tuple of characters ending in </w>.
Count every adjacent pair across the corpus, weighted by word frequency.
Merge the most frequent pair into one symbol — that's one new vocab token.
Repeat num_merges times: common sequences collapse into single tokens.

CPython · WebAssembly

# Byte-Pair Encoding: grow a vocabulary by merging the most
# frequent adjacent pair, over and over. This is the core of
# every modern LLM tokenizer.
from collections import Counter

# corpus as {word: frequency}, each word split into characters + end marker
CORPUS = {
    "low": 5, "lower": 2, "newest": 6, "widest": 3,
}

def get_pairs(vocab):
    pairs = Counter()
    for word, freq in vocab.items():
        syms = word.split()
        for i in range(len(syms) - 1):
            pairs[(syms[i], syms[i + 1])] += freq
    return pairs

def learn_bpe(corpus, num_merges):
    # start every word as space-separated chars + end-of-word marker
    vocab = {" ".join(list(w)) + " </w>": f for w, f in corpus.items()}
    merges = []
    for _ in range(num_merges):
        pairs = get_pairs(vocab)
        if not pairs:
            break
        best = max(pairs, key=pairs.get)      # most frequent pair
        merges.append(best)
        bigram = " ".join(best)
        joined = "".join(best)
        vocab = {w.replace(bigram, joined): f for w, f in vocab.items()}
    return merges

rules = learn_bpe(CORPUS, num_merges=8)
for i, (a, b) in enumerate(rules, 1):
    print(f"merge {i}: ({a!r}, {b!r}) -> {a + b!r}")

QUICK CHECK

Did it stick?

See every answer, with the reasoning

What does one BPE step do?
Answer: Merges the most frequent adjacent pair into a new token — BPE repeatedly finds the most frequent adjacent pair of symbols and merges it into a single new token, adding that merge to the vocabulary.
Why use sub-word tokens instead of whole words?
Answer: They keep the vocabulary small yet never hit an unknown word — Sub-words are the sweet spot: common words become one token, rare words split into reusable pieces, so the vocabulary stays compact and nothing is ever truly unknown.
Why does an LLM struggle to count the r's in "strawberry"?
Answer: It sees tokens, not individual characters — The model never sees letters — the characters are hidden inside tokens, so counting them requires reasoning about spelling it cannot directly observe. It is a tokenization artifact.
If the same idea uses more tokens, what happens?
Answer: It costs more and uses more of the context window — LLMs are billed per token and their context window is measured in tokens, so more tokens for the same meaning means higher cost and less room.

FAQ

Tokenization, answered

What is tokenization in an LLM?

Turning raw text into tokens — usually sub-word chunks — each mapped to an integer ID. The model only ever sees those IDs. Context limits, cost, and some model quirks all trace back to it.

What is Byte-Pair Encoding (BPE)?

The algorithm behind most LLM tokenizers: start from characters, then repeatedly merge the most frequent adjacent pair into a new token. Common words end up as one token; rare words stay split into reusable pieces.

Why not just use words as tokens?

A word vocabulary is huge, chokes on typos, names, and new words (all become 'unknown'), and wastes capacity on run/runs/running. Sub-words are the sweet spot — compact and never truly unknown.

Why does token count matter?

You're billed per token and the context window is measured in tokens. Fewer tokens = cheaper and more room. Rare words, code, numbers, and many languages split into more tokens for the same meaning.

Why do LLMs miscount letters like the r's in 'strawberry'?

The model sees tokens, not letters. The characters are hidden inside tokens, so counting them means reasoning about spelling the model can't directly see — a tokenization artifact, not a reasoning failure.

Finished this one? 0 / 61 Labs done

NextThe Dice Loader: Sampling Next on the “Driving the model” ladder · rung 2 of 4

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering