Codex,
explained.

Rung 1, so the comparison is against how generated code was scored before, which was borrowed from machine translation: measure the text overlap between the generated program and a reference solution. For code that is close to meaningless. Two programs with almost no tokens in common can both be perfectly correct, and a program differing from the reference by one character can be broken.

The change is to run it. A generated program is correct if it passes the unit tests, full stop — functional correctness rather than similarity, evaluated by execution. That sounds obvious and it was the field's methodological turning point for code, because it makes the metric agree with what anyone actually wants. The accompanying benchmark pairs function signatures and docstrings with hidden tests, so a model is asked to implement a specification and is judged on whether the implementation works.

The measurement detail is worth the attention it gets, because it is where naive evaluation goes wrong. What you want to know is whether at least one of k samples solves the problem — that is how a developer uses a code assistant, with several suggestions to pick from. Sampling exactly k and checking is a very high-variance estimate of that. Instead, draw n samples with n larger than k, count how many pass, and compute the probability that a random subset of size k contains a passing one analytically from the count. Same quantity, far less noise, for free — and it is the reason numbers from this benchmark are comparable at all.

Code is text, and a lot of it is public. Codex's recipe was simply to take the GPT architecture and fine-tune it on billions of lines of source code from GitHub. The result is a language model whose "language" was Python source (the paper's models are trained on Python only; the multi-language models behind Copilot are separate) — a model that, given a function signature and a docstring, continues the "sentence" by writing the function body. Because programming languages have tighter structure than natural language, and because so much example code exists, the model became strikingly capable at turning intent into working code.

That capability, wrapped in an editor, became GitHub Copilot — autocomplete that finishes not just the line but the whole function. But shipping a code model raised a sharp evaluation problem the natural-language world had mostly dodged: how do you know if generated code is good? That question is where the paper made its deepest mark.

The old way to score generated text is to compare it to a reference with a similarity metric like BLEU. For code this is badly broken: a correct program can be written a hundred different ways, so a working solution that looks different from the reference scores terribly, while a broken one that looks similar scores well. Text similarity measures the wrong thing entirely.

Codex insisted on functional correctness: a generated program is right if and only if it passes the unit tests. To measure this it introduced HumanEval — 164 hand-written problems, each a function signature, a docstring describing the task, and a set of hidden tests. Generate a solution, run it against the tests, and it counts only if every test passes. Hand-writing the problems (rather than scraping them) was deliberate, to reduce the chance they'd leaked into training data. Grade by execution, not appearance.

Now the subtle part. A code model is a sampler — ask it twice and you get two different programs. So the right question isn't "is its single output correct" but "if it takes k attempts, does at least one work?" That's pass@k: the probability that at least one of k sampled solutions passes the tests. Estimating it naively (sample k, check) is high-variance, so Codex uses an unbiased estimator: sample a larger n, count c that pass, and compute the chance a random k-subset contains no passing one, subtracted from 1:

C(n−c, k)/C(n, k) is the probability that all k drawn solutions come from the failing ones — so 1 minus it is the probability at least one passes. It reduces to the fraction correct at k = 1, hits 1 when every solution passes (or when there are fewer failures than k), and 0 when none pass. Computing it from a big n gives a stable estimate of a small-k quantity. The runnable version below computes the whole pass@k curve.

The pass@k framing exposes a practical truth: more attempts dramatically raise your odds. A model with a modest 30% chance of solving a problem on the first try has a much higher chance that one of ten tries works — because the failures are somewhat independent, ten shots cover far more ground than one.

That gap between "one of these works" and "I know which one" is the crux. High pass@k says the model can produce a solution if you can verify which sample is correct — which for code you often can, by running tests. This is exactly the generate-and-verify asymmetry: generating many candidates is cheap, checking them is cheap (run the tests), so sampling plus a verifier is a powerful recipe. Codex made that concrete for code and standardized how to measure it.

The paper (Chen, Tworek, Jun, Yuan and many colleagues at OpenAI) trains a family of models from 12 million to 12 billion parameters and evaluates them on HumanEval, a benchmark it introduces. The procedure:

1. Collect the code. In May 2020 the team gathered 54 million public GitHub repositories and kept 179 GB of unique Python files under 1 MB. Removing files that looked auto-generated, had an average line length above 100 or a maximum above 1,000, or contained few alphanumeric characters left 159 GB. The corpus is Python only.
2. Adapt the tokenizer. The GPT-3 text tokenizer splits code whitespace badly, so extra tokens for runs of whitespace were added, cutting the number of tokens needed for code by about 30 percent.
3. Fine-tune GPT models on it. Training ran for 100 billion tokens with Adam (β₁ = 0.9, β₂ = 0.95, ε = 10^⁻⁸, weight decay 0.1), a 175-step linear warmup and cosine learning-rate decay. Starting from GPT-3 weights did not improve final performance, presumably because the code corpus is so large, but it converged faster, so the models were initialized from GPT-3 anyway.
4. Prompt with a specification. Each HumanEval prompt is a function header and docstring, sometimes with imports or helper functions; the model writes the body.
5. Sample and stop. Completions are drawn with nucleus sampling (top p = 0.95) and cut at the first stop sequence, such as a new top-level def, class, if, print or comment line, so the model cannot carry on past the function.
6. Execute in a sandbox. Generated programs run in a gVisor-based sandbox with no network access, because the code is untrusted by construction.
7. Estimate pass@k without bias. For each problem, n = 200 samples are drawn and the c that pass every unit test are counted; pass@k for k up to 100 comes from the unbiased estimator below, averaged over problems.
8. Codex-S: fine-tune on standalone functions. A second training set was built in HumanEval's format: about 10,000 problems from competitive-programming sites and about 40,000 from continuous-integration runs of open-source Python packages, where the inputs and outputs of functions were traced while the projects' own tests ran. Problems Codex-12B could not solve in 100 samples were dropped as either ambiguous or too difficult.
9. Rank without tests. When only one sample can be shown and no tests are available, the paper picks the sample with the highest mean token log-probability.

Temperature is part of the method

Because pass@k rewards diversity, the best sampling temperature depends on k. The paper swept it and found T = 0.2 best for pass@1 and T = 0.8 best for pass@100: one attempt wants the single most likely program, while a hundred attempts want programs that differ from one another. The same sweep is why a pass@k number is incomplete without the temperature it was measured at. A third variant, Codex-D, was trained in the reverse direction, writing a docstring from a function body, and because no unit test can check a docstring, its outputs were graded by hand.

• n — samples drawn per problem (200 in the paper); c — how many pass every test; k — attempts allowed.
• C(n − c, k)/C(n, k) — the chance that k samples drawn without replacement from the n are all failures.
• Π(1 − k/i) — the same ratio as a product of c terms, which avoids enormous binomial coefficients; the paper's reference code computes it this way.
• 𝔼problems — the estimate is made per problem and then averaged, so pass@k depends on how solve rates are spread across problems, not only on their mean.

Plugging the observed rate c/n into 1 − (1 − p)^^k looks natural but underestimates: that function is concave in p, so its average over the noise in c/n is smaller than its value at the true rate. At the extremes the unbiased estimator gives c/n at k = 1, exactly 1 when fewer than k samples fail (n − c < k), and 0 when nothing passes.

The headline evaluation is HumanEval, 164 hand-written Python problems with an average of 7.7 unit tests each, scored with n = 200 samples per problem. The comparison models were GPT-3, which had no code-specific training, and the open GPT-Neo and GPT-J models, trained on the Pile, which includes GitHub code.

pass@1 rose smoothly with model size across the family. The gap between pass@1 and pass@100 is the paper's case for sampling plus a selection step: with an oracle that runs the hidden tests, the 12B model solves about 70 percent of problems from 100 samples, against 28.8 percent from one. Ranking by mean log-probability recovers part of that gap without any tests, while ranking by the summed log-probability did slightly worse than picking at random.

The negative results are as useful as the headline. BLEU scores of correct and incorrect solutions overlapped heavily, so a text-similarity metric could not tell them apart. Performance fell sharply as docstrings chained more operations together, roughly exponentially in the number of steps, which the paper reads as a weakness in binding many operations to variables. And when prompts contained subtle bugs, the models wrote worse code than they were capable of, a misalignment that grew with model size.

The benchmark's narrowness is acknowledged: self-contained functions with clear docstrings and a handful of tests are not what software engineering is. There is no repository to navigate, no existing code to avoid breaking, no ambiguity in the specification. It measures a real skill and a small one.

The paper is also unusually direct about the problems that come with shipping this, and the list has aged well: the model produces code that is subtly wrong in plausible ways, can suggest insecure patterns because insecure patterns are abundant in the training data, may reproduce training material, and invites over-reliance from users who read fluent code as correct code.

How it held up: functional correctness won so completely that the alternative is unthinkable, and the unbiased estimator is standard. The benchmark itself became saturated and contaminated — it is small, public, and years old, so top scores now say little and the results cannot be cleanly separated from exposure during pretraining. That trajectory is the normal life of a good benchmark, and rung 3 exists largely because of it.

Codex kicked off the code-model era: a lineage of ever-stronger coding models, richer benchmarks (MBPP, SWE-bench, and beyond), and eventually the coding agents that don't just autocomplete but plan, edit, run, and debug across a repository. The through-line from Codex is the pairing it made standard — a generative model plus an executable check.

That pairing is its deepest legacy. Code is special among AI tasks because correctness is mechanically verifiable — you can just run it. Codex leaned into that: measure by execution (pass@k), and exploit it (sample many, keep what passes). The same generate-and-verify structure now underlies coding agents that iterate against tests and reasoning systems trained on verified answers. Codex was where "let the machine write it, then check it actually works" became the organizing principle of AI for code.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Completion, then competition, then a benchmark made of real GitHub issues — the point at which "can it code?" got an answer you could audit.

1. 1Codex2021
2. 2AlphaCode2022
3. 3SWE-bench & SWE-agent2023

All 110 papers, in 19 ladders →