Tokenizer efficiency, actually measured

You are billed per token, your context window is measured in tokens, and your latency scales with tokens. But a tokenizer doesn't turn text into tokens at a fixed rate — a "token" is a learned sub-word unit, and how many of them a given string becomes depends on how well that string matches the vocabulary the tokenizer was trained on. Two tokenizers handed the exact same document can differ by 2× or more in the token count they produce, which is a 2× difference in cost and context consumption for zero difference in input.

The usual way this gets summarized — "chars per token," higher is more efficient — is directionally right and hides a real trap that this benchmark is specifically built to expose.

Nine production tokenizers were loaded and run against three corpora: a passage of English prose about the history of AI, a block of real Python code, and a passage of non-English text (Hindi and Chinese). For each tokenizer × corpus pair, three things were measured:

Characters per token — the corpus's character count divided by the token count it produced. Higher means fewer tokens for the same text, i.e. cheaper and more context-efficient. Encode throughput — best-of-20 wall-clock characters-per-second, to compare raw tokenization speed. Fidelity — the fraction of tokens that are the "unknown" token, and whether decode(encode(text)) reproduces the original's length. This last metric is the one that makes the benchmark honest.

On ordinary English prose, all nine tokenizers land within a narrow band of roughly 4.5 to 5.2 characters per token. This is the case everyone's intuition is calibrated on, and it's the least interesting one — English is what every one of these tokenizers was most heavily trained on, so they all handle it competently.

Feed the same nine tokenizers Python code, and then non-English text, and the tight English cluster falls apart. GPT-2, the oldest tokenizer here, needs 2.3 characters per token on code — it fragments code into more than twice as many tokens as cl100k_base (4.21 c/t) does, because its older vocabulary never learned the common multi-character patterns of code. On non-English text the gaps get even larger, and this is where the fidelity check becomes essential.

Look at the non-English column again. By raw characters-per-token, T5 appears to be the single most efficient tokenizer on non-English text — 3.39 chars/token, comfortably ahead of everything else. A benchmark that stopped at chars-per-token would rank it first and publish that as a finding. It would be completely wrong.

The fidelity check tells the real story: 48% of T5's tokens on that text are the <unk> (unknown) token, and decode(encode(text)) reproduces only 17% of the original's length. T5's SentencePiece vocabulary was trained on English and simply cannot represent Hindi or Chinese characters — so instead of tokenizing them, it collapses whole runs of text into a single "I don't know this" token. That's why its token count is so low: it isn't efficiently encoding the text, it's throwing most of the text away. The impressive-looking 3.39 chars/token is the signature of data loss, not efficiency. BERT-base-uncased (English-only) shows a milder version of the same failure: 36% unknown tokens on the same input.

Efficiency isn't only about token count — how fast a tokenizer runs matters for high-throughput pipelines that tokenize billions of characters. Here the split is stark and has a clear cause: OpenAI's tiktoken tokenizers (cl100k_base, o200k_base) are implemented in optimized Rust, while the HuggingFace-loaded tokenizers, though also Rust-backed, carry more per-call overhead in this setup. On English prose, o200k_base sustained ~32.7M characters/second and cl100k_base ~20.9M/s, while the rest clustered around 4–5M/s.

Every number on this page comes from the script below. The token counts are deterministic — you'll get the identical characters-per-token and identical unknown-token ratios on any machine. Only the speed numbers depend on your hardware. Install tiktoken and tokenizers, then run this against the same three corpora:

# pip install tiktoken tokenizers
# For each tokenizer, on each corpus, we measure:
#   - token_count and chars_per_token  (deterministic, reproduces anywhere)
#   - unk_ratio: fraction of tokens that are the <unk> / [UNK] token
#   - roundtrip_len_ratio: len(decode(encode(text))) / len(text)
#     -> anything well below 1.0 means the tokenizer is DROPPING characters
#        it can't represent, which inflates its apparent efficiency.

import tiktoken
from tokenizers import Tokenizer

def measure(encode, decode, unk_id, text):
    ids = encode(text)
    n = len(ids)
    unk = ids.count(unk_id) if unk_id is not None else 0
    decoded = decode(ids)
    return {
        "chars_per_token":     len(text) / n,
        "unk_ratio":           unk / n,
        "roundtrip_len_ratio": len(decoded) / len(text),   # << the fidelity check
    }

# tiktoken (byte-level BPE: no <unk>, always round-trips faithfully)
enc = tiktoken.get_encoding("o200k_base")
print(measure(lambda t: enc.encode(t, disallowed_special=()),
              enc.decode, None, YOUR_TEXT))

# HuggingFace tokenizers (may have <unk>; CHECK roundtrip_len_ratio)
tok = Tokenizer.from_pretrained("t5-base")
unk = tok.token_to_id("<unk>")
print(measure(lambda t: tok.encode(t).ids, tok.decode, unk, YOUR_NON_ENGLISH_TEXT))
# -> t5-base on Hindi/Chinese: chars_per_token LOOKS great (~3.4),
#    but unk_ratio ~0.48 and roundtrip_len_ratio ~0.17 expose that it is
#    discarding ~half the text. Efficiency by token count alone is a lie here.