Codex,
explained.

Code is text, and a lot of it is public. Codex's recipe was simply to take the GPT architecture and fine-tune it on billions of lines of source code from GitHub. The result is a language model whose "language" includes Python, JavaScript, and dozens of others — a model that, given a function signature and a docstring, continues the "sentence" by writing the function body. Because programming languages have tighter structure than natural language, and because so much example code exists, the model became strikingly capable at turning intent into working code.

That capability, wrapped in an editor, became GitHub Copilot — autocomplete that finishes not just the line but the whole function. But shipping a code model raised a sharp evaluation problem the natural-language world had mostly dodged: how do you know if generated code is good? That question is where the paper made its deepest mark.

The old way to score generated text is to compare it to a reference with a similarity metric like BLEU. For code this is badly broken: a correct program can be written a hundred different ways, so a working solution that looks different from the reference scores terribly, while a broken one that looks similar scores well. Text similarity measures the wrong thing entirely.

Codex insisted on functional correctness: a generated program is right if and only if it passes the unit tests. To measure this it introduced HumanEval — 164 hand-written problems, each a function signature, a docstring describing the task, and a set of hidden tests. Generate a solution, run it against the tests, and it counts only if every test passes. Hand-writing the problems (rather than scraping them) was deliberate, to reduce the chance they'd leaked into training data. Grade by execution, not appearance.

Now the subtle part. A code model is a sampler — ask it twice and you get two different programs. So the right question isn't "is its single output correct" but "if it takes k attempts, does at least one work?" That's pass@k: the probability that at least one of k sampled solutions passes the tests. Estimating it naively (sample k, check) is high-variance, so Codex uses an unbiased estimator: sample a larger n, count c that pass, and compute the chance a random k-subset contains no passing one, subtracted from 1:

C(n−c, k)/C(n, k) is the probability that all k drawn solutions come from the failing ones — so 1 minus it is the probability at least one passes. It reduces to the fraction correct at k = 1, hits 1 when every solution passes (or when there are fewer failures than k), and 0 when none pass. Computing it from a big n gives a stable estimate of a small-k quantity. The runnable version below computes the whole pass@k curve.

The pass@k framing exposes a practical truth: more attempts dramatically raise your odds. A model with a modest 30% chance of solving a problem on the first try has a much higher chance that one of ten tries works — because the failures are somewhat independent, ten shots cover far more ground than one.

That gap between "one of these works" and "I know which one" is the crux. High pass@k says the model can produce a solution if you can verify which sample is correct — which for code you often can, by running tests. This is exactly the generate-and-verify asymmetry: generating many candidates is cheap, checking them is cheap (run the tests), so sampling plus a verifier is a powerful recipe. Codex made that concrete for code and standardized how to measure it.

Codex's headline numbers were strong for 2021 — solving a meaningful fraction of HumanEval on the first try and much more with sampling — but its real impact was in production:

The productization mattered as much as the research: Copilot made everyday developers' first real contact with a large language model, years before ChatGPT, normalizing the idea of AI as a coding collaborator. And the evaluation contribution outlived the specific model — every code model is still reported in pass@k on HumanEval-style benchmarks.

Codex kicked off the code-model era: a lineage of ever-stronger coding models, richer benchmarks (MBPP, SWE-bench, and beyond), and eventually the coding agents that don't just autocomplete but plan, edit, run, and debug across a repository. The through-line from Codex is the pairing it made standard — a generative model plus an executable check.

That pairing is its deepest legacy. Code is special among AI tasks because correctness is mechanically verifiable — you can just run it. Codex leaned into that: measure by execution (pass@k), and exploit it (sample many, keep what passes). The same generate-and-verify structure now underlies coding agents that iterate against tests and reasoning systems trained on verified answers. Codex was where "let the machine write it, then check it actually works" became the organizing principle of AI for code.