AlphaCode,
explained.

Rung 1 samples a handful of candidates and reports whether one works. This rung takes that observation to its limit: if the model solves a problem some small fraction of the time, then sampling enormously and selecting well should solve problems it cannot solve reliably at all. The target is competitive programming — problems requiring an actual algorithmic insight, where rung 1's models were near-hopeless.

The pipeline is generate, filter, cluster. Generate candidate programs on the order of a million per problem. Then filter on the example tests the problem statement provides, which eliminates the overwhelming majority — this is the step that makes the scale usable, and it is free, because the examples come with the question.

The survivors are still far too many to submit, since contests limit attempts, and the clever part is how they are reduced. Cluster by behaviour, not by text: generate additional inputs, run every surviving program on them, and group programs that produce identical outputs. Each cluster is one distinct solution strategy, however differently written, so submitting one representative per cluster covers the space of genuinely different attempts rather than submitting ten spellings of the same wrong idea. Stated in units: reaching roughly median human in real contests, from a model that gets any individual sample right very rarely.

Competitive programming problems aren't like the tidy functions of HumanEval. They're puzzles: a clever problem statement that hides an algorithmic insight, a strict time and memory limit, and hidden tests that punish any edge case you miss. Solving them requires genuine reasoning — pick the right algorithm, get the complexity under budget, implement it without bugs. For a long time this looked like a stronghold of human ingenuity.

AlphaCode's insight was that you don't need a model that gets it right on the first try — you need one that gets it right somewhere among many tries, plus a reliable way to find that right answer in the pile. So it split the job into two halves: a model that generates a huge, diverse set of candidate programs, and a selection pipeline that separates the wheat from the chaff. Brilliance in aggregate, not per sample.

The first half is a large transformer trained on public code and fine-tuned on competitive-programming problems, used to sample candidate solutions — but at a scale most systems never attempt. For a single hard problem, AlphaCode generates up to millions of candidate programs, using high sampling temperature and metadata conditioning to keep them diverse.

Why so many? Because for a genuinely hard problem, the per-sample probability of being correct is tiny. But "tiny per sample" across a million samples still yields a healthy number of correct ones — if some correct solution is within the model's reach at all. Massive sampling is a way of buying coverage of the solution space. The whole strategy rests on it: you can afford to be wrong nearly every time, as long as you're right often enough in absolute terms and can identify the winners.

A million candidates is far too many to submit — and contests punish wrong submissions and cap how many you can make. So the pile must be cut down, hard. The first cut is filtering: every problem comes with a few public example tests, so AlphaCode runs each candidate on those and throws away any that fail. This is cheap (just execution) and brutally effective — it eliminates the large majority, often ~99%, of the samples, because a program that can't even pass the given examples is certainly wrong.

Filtering is the workhorse that turns an unmanageable flood into a manageable shortlist. It costs almost nothing — no model, just running code against a handful of tests — yet removes nearly everything wrong. What's left has cleared the lowest bar (it works on the examples), but there are still far too many to submit, and many are duplicates. That's where clustering comes in. The runnable version below filters candidates and shows only the passers survive.

The thousands of surviving programs aren't thousands of distinct solutions — many are syntactic variations of the same underlying algorithm. AlphaCode clusters them by behavior: it generates additional test inputs (with a separate model), runs every survivor on them, and groups programs that produce identical outputs. Programs in the same cluster are effectively the same solution; different clusters are genuinely different attempts.

Then it submits one program per cluster, prioritizing the largest clusters, until it hits the submission budget. The logic is a form of consensus: if many independently-sampled programs converge on the same behavior, that behavior is more likely to be correct than a lone outlier. Clustering both deduplicates (don't waste submissions on the same solution) and ranks (agreement is a signal). It's the same "sample many, take the majority" idea that powers self-consistency in reasoning — here applied to whole programs judged by what they compute.

AlphaCode (Li, Choi, Chung, Kushman and colleagues at DeepMind) is a pipeline, not just a model. In order:

1. Pre-train on GitHub. The pre-training set is a snapshot of public GitHub code in a dozen popular languages, including C++, Python and Java, about 715 GB after deduplication and filtering. Each file is split at a random point: the encoder reads the first part and the decoder is trained to produce the rest, with a masked-language-model loss on the encoder as well.
2. Use an asymmetric encoder-decoder. Problem statements are long and solutions shorter, so the encoder takes 1,536 tokens and the decoder 768. The encoder is shallow and the decoder deep, and attention uses multi-query attention, in which all heads share one set of keys and values, to make sampling cheaper. Models range from 300M to 41B parameters, with an 8,000-token SentencePiece vocabulary shared across programming languages and English.
3. Fine-tune on CodeContests. The team built and released this dataset: problems from Codeforces and other sources, with correct and incorrect human submissions, and a strict temporal split so that every validation and test problem appeared after the training data was collected. Extra tests were generated by mutating existing test inputs and keeping those on which correct solutions agree, to cut false positives.
4. Condition on metadata. The fine-tuning prompt includes the problem's difficulty rating, its algorithm tags, the language, and whether the solution is correct. At sampling time the model is always told the solution is correct, and the rating and tags are randomized to make samples more diverse.
5. Train with GOLD and tempering. Because a contest needs only one correct solution per problem, the loss uses GOLD, an off-policy objective that weights each token by the model's own probability of it, so the model concentrates on solutions it already finds likely rather than spreading itself over all of them. Logits are divided by a temperature of 0.2 during training, which smooths the learned distribution; sampling uses a separately tuned temperature.
6. Sample at scale. For each problem the models sample on the order of a million programs, split between Python and C++.
7. Filter on the examples. Every sample runs on the example tests given in the problem statement; about 99 percent fail and are discarded.
8. Cluster by behavior. A separate model, trained to write plausible test inputs from a problem description, generates new inputs. The survivors run on them and are grouped by identical outputs.
9. Submit ten. One program is taken from each cluster, largest cluster first, until the limit of 10 submissions per problem is reached.

Why each stage is there

Pre-training gives general code knowledge, fine-tuning gives the contest format, and the sampling tricks (metadata randomization, high-volume sampling in two languages) buy diversity, because coverage of genuinely different attempts is what the later stages need. Filtering is nearly free and removes almost everything; clustering exists because the few thousand survivors are mostly the same handful of behaviors written differently, and ten submissions cannot be spent on near-duplicates.

• k — samples generated per problem; n — submissions allowed, 10 in AlphaCode's main metric. A problem counts as solved if any submitted program passes all hidden tests.
• pass@k — the special case where every sample may be submitted, so it upper-bounds 10@k at the same k; the gap between them measures how good the selection is.
• zt, T — the decoder's logits and the training temperature; because training divides by 0.2, the raw logits the model learns are flatter than the data.
• P̂θ(yt) — the model's own probability of the demonstrated token, used as a constant weight with no gradient flowing through it.

At the extremes: n = 1 is the hardest version, a single submission, and n = k is pass@k. If the GOLD weight were replaced by 1, the objective would be ordinary maximum likelihood, which tries to cover every human solution; with the weight, solutions the model finds unlikely are down-weighted, trading recall for precision, which is what a contest that needs one correct program rewards.

AlphaCode was evaluated two ways: on 10 recent Codeforces contests, each with more than 5,000 participants, by simulating participation with up to 10 submissions per problem under the contest's penalty rules, and on the held-out CodeContests test set with the n@k metric.

A ranking in the top 54.3 percent places AlphaCode near the median contestant, slightly below it. The ablations credit every part of the recipe: each fine-tuning addition (metadata conditioning, value conditioning, GOLD, tempering) raised solve rates at large sample budgets, larger models helped, and more samples helped at every model size. The scaling result is also the method's main limit: solve rate grew roughly with the logarithm of the sample count, so each further gain needed ten times the compute.

The dataset work exposed a problem in earlier benchmarks. With only a few tests per problem, many programs that passed were actually wrong; the authors report high false-positive rates in existing datasets and say their generated tests brought CodeContests down to a few percent. They also checked for copying and found that AlphaCode's solutions shared substrings with the training data at about the same rate as human solutions, mostly input-parsing boilerplate rather than problem-solving logic. Solve rates fell steeply with problem difficulty, and the hardest problems stayed out of reach.

DeepMind's AlphaCode 2 (2023), built on Gemini, kept the sample, filter and cluster skeleton, added a learned model to score candidates, and is reported to perform better than about 85 percent of contest participants.

The compute cost is stated and it is extraordinary — a million samples per problem is a quantity of inference nobody would spend on a real task, so this is a demonstration of what is possible rather than a method. It also depends on example tests being supplied with the problem, which is a property of programming contests and not of programming.

And competitive programming is a narrow, self-contained domain: a precise specification, a closed input space, a definite answer. Success here transfers less than the framing suggests, which the paper is more careful about than its coverage was.

How it held up: the recipe is the ancestor of test-time compute for code, and behavioural clustering is the durable idea — semantic deduplication by observed output is cheap, does not care how the program is written, and has no equivalent in text. What did not hold is the economics: within a couple of years, reasoning models reached comparable and better contest performance with a tiny fraction of the samples, because a model that reasons before answering does not need a million attempts to stumble on the insight. Read this rung as the strongest available demonstration that search over samples and reasoning within a sample are substitutes, and that the second is far cheaper once available.

AlphaCode is a landmark case of test-time scaling for code: rather than a smarter single generation, throw compute at many generations and select well. That pattern — sample-and-select — recurs across the field, from self-consistency in reasoning to best-of-N with a verifier, and it foreshadowed the reasoning models that spend heavy compute exploring many solution paths before answering. AlphaCode made the case, on a hard task, that quantity plus good selection can substitute for one-shot brilliance.

Its deeper idea is the generate-and-verify asymmetry taken to an extreme. Generating a correct competitive-programming solution is very hard; checking whether a program passes tests is trivial. AlphaCode leaned all the way into that gap: generate absurdly many, let cheap execution filter, and let behavioral agreement rank. When you can verify far more cheaply than you can produce, flooding the problem with attempts and sieving them is a startlingly effective strategy — one AlphaCode proved could reach human-competitive performance on a task that seemed to demand genius.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Completion, then competition, then a benchmark made of real GitHub issues — the point at which "can it code?" got an answer you could audit.

1. 1Codex2021
2. 2AlphaCode2022
3. 3SWE-bench & SWE-agent2023

All 110 papers, in 19 ladders →