AI & LLMs

Token

The unit a model actually reads and writes — a sub-word chunk, not a character or a word.

A token is a piece of text produced by a tokenizer, typically a common sub-word (“play”, “‑ing”). Models process tokens, price by tokens, and are bounded by a token context window. This is why models miscount letters in a word: they never see the letters, only the token.