Text becomes
meaning.

A compiler turns source text into something runnable, and the secret is that it never does it all at once — it's a pipeline of stages, each with one clear job. First, tokenizing (lexing) chops the raw characters into meaningful atoms: numbers, operators, identifiers, keywords. Then parsing arranges those tokens into a structure that captures the grammar — crucially, which operations bind to which (precedence). Then evaluation or code generation turns that structure into a result or into machine code.

An interpreter shares the same front end — tokenize, parse — but evaluates the structure directly instead of emitting code; a compiler emits code to run later. Either way, the shape is identical: text → tokens → structure → meaning, one well-defined transformation at a time. That staging is what makes something as daunting as "understand a programming language" tractable: no single step is hard, and each hands a cleaner representation to the next. We'll build the smallest real instance — an arithmetic calculator that respects precedence — because it contains the whole idea in miniature.

The first stage, lexical analysis, scans the character stream and groups characters into tokens — the words of the language. Given "12 + 3", the lexer emits [12, +, 3]: it recognizes that the adjacent digits 1 and 2 form one number token, treats + as an operator token, and throws away the whitespace. It turns a flat, characterwise string into a clean sequence of typed units.

This matters because it frees every later stage from dealing with raw characters. The parser doesn't want to worry about whether a number is one digit or seven, where spaces go, or how to skip a comment — the lexer has already resolved all of that, handing up a tidy token stream. In a real language the lexer also recognizes string literals, identifiers, keywords versus names, and multi-character operators like == or ->. But the essence is exactly the calculator's: read left to right, accumulate characters into the token they belong to, and emit clean atoms. Get the atoms right and the hard part — structure — has something orderly to work on.

Now the crux: 2 + 3 * 4 must mean 2 + (3 * 4), not (2 + 3) * 4. Encoding that is the parser's job, and the classic compact way is Dijkstra's shunting-yard algorithm, which converts our normal infix notation into Reverse Polish Notation (RPN, postfix) — a form with no parentheses and no ambiguity, trivial to evaluate.

The precedence lives in one comparison: when * is on the stack and + arrives, * (higher precedence) is popped to output first, so it will be applied before +. That single rule is where "multiplication binds tighter" is encoded. Parentheses override it by forcing everything inside to flush before the closing ). The output, RPN, is then evaluated with a plain stack: push numbers, and on each operator pop the top two, apply, push the result. All the precedence complexity moved into the parse; evaluation is dead simple. (Real compilers usually build an explicit tree — an AST — but RPN captures the same precedence resolution in the clearest possible form.)

The whole calculator is the composition of three functions — the classic compiler pipeline in miniature:

tokenize: string → tokens. to_rpn: infix tokens → postfix (shunting-yard). eval_rpn: postfix → value (stack).

Shunting-yard's precedence rule is the single comparison that makes * bind before +, and stack evaluation folds the RPN into a number:

Because 3*4 is computed before the +, the answer is 14; parentheses (2+3)*4 flush the + first and give 20. The runnable version below is the whole tokenize → RPN → eval pipeline.

The calculator is small, but the skeleton is exactly what a production compiler uses — just with more stages bolted on:

The shape never changes: text becomes tokens becomes a tree becomes lower-level code. A modern compiler adds an intermediate representation (IR) in the middle so optimization and code generation can target many source languages and many CPUs through a shared core (that's the whole idea behind LLVM). And an interpreter simply stops after building the tree and walks it. Even the thing turning your prompts into tokens for a language model is a lexer at heart. Once you see the pipeline in the calculator, you see it everywhere — the difference between a toy and a real compiler is the number of stages and the richness of each, not the shape.

The first classic mistake is parsing with regexes or string splitting. It works for the simplest inputs and then collapses the moment you hit nesting — parentheses inside parentheses, precedence, function calls. Regular expressions genuinely cannot parse a nested grammar (it's a formal-language fact, not a skill issue); you need a real parser with a stack or recursion. Reaching for str.split to "parse" structured input is a trap that always ends in a rewrite.

Two more. Conflating the stages: trying to tokenize and evaluate in one pass tangles concerns and makes precedence and errors nearly impossible to get right — keep lex, parse, and evaluate as separate, testable steps, exactly as the pipeline prescribes. And ignoring error handling: real input is malformed — unmatched parentheses, stray operators, unexpected end — and a parser must report where and why, not crash or silently produce garbage (this is why lexers track line and column positions). The payoff for respecting the pipeline is huge: parsing and interpreting stop being intimidating and become a sequence of small, well-understood transformations you can build and test one at a time. The whole handbook is that realization — a compiler is not magic, it's a pipeline: chop into tokens, reorder by precedence, evaluate on a stack, and repeat the pattern all the way up to a real language.