GPT-3,
explained.

Rungs 1 and 2 both end with a gradient step on your task. BERT fine-tunes a head; T5 fine-tunes the whole model on a task expressed as text. GPT-3's claim is that at sufficient scale you can skip that entirely: put a few worked examples in the prompt, and the model performs the task without any weight update at all. No labelled training set, no optimiser, no checkpoint per task.

In the ladder's units: T5's largest was 11B parameters; GPT-3 is 175B, a ~16× jump, trained on roughly 300B tokens. Hold those two numbers together, because rungs 4 and 5 are an argument about precisely that ratio: 175B parameters against 300B tokens is about 1.7 training tokens per parameter. Nothing in this paper suggests that ratio is a choice at all — it reads as a consequence of using the compute available.

The interface change is the lasting one. Once the way to get behaviour out of a model is to write text at it, prompting becomes an engineering discipline, and everything on the reasoning ladder becomes possible.

BERT had made a recipe famous: pre-train a big model once, then fine-tune it on each specific task with a labeled dataset. It worked brilliantly — but it had a cost that was easy to overlook. Every new task needed its own curated collection of thousands of labeled examples, and its own round of training that nudged the model's weights.

Humans don't work that way. You can be told "translate these to French" and shown one or two examples, and you're off — no dataset of ten thousand pairs, no retraining of your brain. The GPT-3 authors framed this squarely: the fine-tuning recipe was powerful but narrow, and it scaled poorly to the long tail of tasks nobody had built a dataset for. They wanted a model that could pick up a task the way a person does — from a short description and a handful of examples.

GPT-3's architecture is almost boringly familiar: it's the same autoregressive Transformer decoder as GPT-2, trained on the same humble objective — predict the next word, over and over, across a huge slice of the internet. The paper introduces essentially no new architecture. Its one radical move is scale.

GPT-2 had 1.5 billion parameters. GPT-3 has 175 billion — over a hundred times larger — trained on roughly 300 billion tokens of text. The hypothesis was that capability wasn't waiting on a cleverer design but on size, and that at some scale, new behaviors would simply emerge. The title says it all: Language Models are Few-Shot Learners. Scale was the whole idea.

The paper's central move is to stop fine-tuning entirely and instead put the "training" inside the prompt. It measures three settings, all with the model's weights frozen:

Zero-shot — you give only a task description. One-shot — you add a single worked example. Few-shot — you add several (often 10–100) examples before the real question. In every case the model just reads the prompt and continues it.

Nothing here updates a single weight. The examples are just text the model reads on its way to generating the answer. And yet, at 175B, accuracy climbs steadily as you go from zero- to one- to few-shot — the extra examples genuinely help. Smaller models barely benefit; the effect grows with scale.

GPT-3 is a stack of 96 Transformer-decoder layers, 12,288-dimensional embeddings, 96 attention heads, and a context window of 2,048 tokens. Each layer is the standard decoder block: masked self-attention (each token attends only to tokens before it) followed by a feed-forward network. Trained on next-token prediction, it becomes a probability machine — given everything so far, what word comes next?

That single objective is deceptively powerful. To predict the next word well across the whole internet, the model has to implicitly pick up grammar, facts, translation, arithmetic, and reasoning patterns — because all of those show up in text. Few-shot prompting simply conditions that machine: the examples in the prompt shift the probabilities so the most likely continuation is the answer in the format you demonstrated.

"175 billion parameters" sounds like a press-release number, but you can compute it from the architecture on a napkin. A Transformer layer's weights are dominated by four attention projections (each d×d) and two feed-forward matrices (d×4d and 4d×d), giving ≈ 12·d² parameters per layer:

The same formula reproduces the whole model family the paper trained (embedding matrix ≈ 50257·d added; it matters for the small ones, vanishes at scale):

Training cost falls out just as cleanly. Each token requires roughly 6 FLOPs per parameter (2 for the forward pass, 4 for backward), and GPT-3 saw 300B tokens:

That's several thousand petaflop/s-days. These two rules — 12·L·d² and 6·N·D — are the napkin math practitioners still use to size and price every model since.

The headline phenomenon has a name: in-context learning. The model appears to "learn" the task from the examples in its context window, during a single forward pass, with no gradient descent and no weight change anywhere. Whatever adaptation is happening lives entirely in the activations as the model processes the prompt.

This was genuinely unexpected. Learning, in machine learning, had always meant updating weights. GPT-3 showed a second, temporary kind of learning that happens at inference time, purely from what's in front of the model. It reframed the interface to AI: you no longer train a model for a task, you describe the task and show examples. That's the birth of prompting as a discipline.

GPT-3 introduces almost no new architecture, so the method is mostly in the data, the training run and the way tasks are posed at test time.

1. Filter Common Crawl with a classifier. Forty-one monthly Common Crawl shards from 2016 to 2019, about 45 TB of compressed plaintext, were scored by a logistic-regression classifier trained to tell curated text (WebText, Wikipedia and a web-books corpus, as positives) from raw Common Crawl (negatives), using Spark's standard tokenizer and hashed term-frequency features. A document was kept when a random Pareto draw with shape α = 9 exceeded one minus its score, so high-scoring pages usually survive and a minority of low-scoring pages still get through, for diversity.
2. Deduplicate. Fuzzy, document-level deduplication within and across sources, using MinHash locality-sensitive hashing (Spark's MinHashLSH with 10 hashes) on the same features, cut the data by about 10%. What remained of Common Crawl was 570 GB, roughly 410B tokens.
3. Weight the mix by quality, not by size. Training draws from five sources at fixed rates, shown in the figure. Sources judged higher quality are sampled more often than their size alone would give, so Wikipedia and WebText2 are seen roughly three times while Common Crawl and Books2 are seen less than half a time: a deliberate trade of a little overfitting for a higher share of good text.
4. Tokenize and pack. GPT-2's byte-level BPE, with a vocabulary of 50,257. Every training sequence fills the full 2,048-token context; shorter documents are packed together, separated by an end-of-text token, with no special masking between them.
5. Use GPT-2's architecture, with sparse layers. A decoder-only Transformer with pre-normalization and GPT-2's modified initialization, except that layers alternate between dense attention and locally banded sparse attention, in the style of the Sparse Transformer. Eight sizes were trained, from 125M to 175B parameters, all on the same 300B tokens.
6. Optimize. Adam with β₁ = 0.9, β₂ = 0.95 and ε = 10^⁻⁸; the gradient norm is clipped at 1.0; weight decay is 0.1. The learning rate warms up linearly over the first 375M tokens, then follows a cosine decay to 10% of its peak over 260B tokens. The batch grows linearly from 32k tokens to full size over the first 4 to 12B tokens; for the 175B model the full batch is 3.2M tokens and the peak learning rate 0.6 × 10^⁻⁴. Training ran on V100 GPUs in a Microsoft cluster, with each model split across GPUs both within each matrix multiply and across layers.
7. Try to decontaminate. Overlaps with benchmark test sets were searched for and removed before training, but a bug left some in, and the run was too expensive to repeat. The paper measures the damage afterwards instead.
8. Evaluate in context. For each test example, K demonstrations are drawn at random from the task's training set and placed before it, separated by one or two newlines, with K anywhere from 0 to as many as fit in 2,048 tokens, typically 10 to 100. Multiple-choice tasks compare the model's likelihood of each answer; free-form tasks use beam search with width 4 and a length penalty of 0.6. No weights change at any point.

• xt — the next BPE token; the context is at most 2,048 tokens. This is the only training objective.
• demos — K solved examples placed in the prompt; query — the unsolved one. θ is identical for every task: demonstrations enter only through what the model is conditioned on.
• c — a candidate answer; 1/|c| — per-token normalization, so a long answer is not penalized merely for having more tokens.
• The third form, used on a few datasets such as ARC, OpenBookQA and RACE, divides by the answer's probability after a bare Answer: prompt. An option that is likely in general, such as a short common word, then gets no credit unless the question itself raises its probability.

At the extremes: with K = 0 the model sees only a natural-language task description, which is zero-shot. As K grows the prompt fills the 2,048-token window, which is the hard ceiling on few-shot learning. Nothing in either scoring rule involves a gradient.

All eight sizes were evaluated with frozen weights on more than two dozen datasets in zero-, one- and few-shot settings. The comparison points are fine-tuned systems that trained on thousands of labeled examples.

Few-shot GPT-3 was often competitive with fine-tuned models and sometimes matched them; zero-shot alone beat the LAMBADA state of the art by about 8 points. It also translated with no fine-tuning on parallel text, unscrambled words and answered trivia closed-book. The trend mattered more than any single score: accuracy rose with model size in every setting, and the gap between few-shot and zero-shot widened with scale, which is the evidence that in-context learning is something larger models grow into.

The negative results are as informative. Tasks that compare two sentences stayed weak: WiC sat at chance, and ANLI was near chance for all but the largest model. Arithmetic decayed quickly with more digits, and a search of the training data for the 3-digit problems found few matches, suggesting the successes were not recall. The contamination study rebuilt each benchmark without any test example sharing a 13-gram with the training data; for most benchmarks the clean scores barely moved, and a few results, including PIQA and Winograd, were flagged.

The paper's own limitations section is longer and franker than most: few-shot performance is still well short of fine-tuned state-of-the-art on many tasks; the model is weak on tasks requiring comparison of two sentences; it has no notion of which of its outputs are true; and the authors flag both the training-data contamination problem and the energy cost of the run. They also state plainly that the model reproduces the biases of its corpus.

The admission most relevant to this ladder is quieter: they note the model is trained once, and that they could not afford to retrain it to explore alternatives. A single un-repeatable run is not a measurement of how to spend compute — which is the gap rung 4 sets out to fill.

How it held up: in-context learning held up completely and is now the primary interface to every frontier model. The specific sizing did not. Within two years the same compute was understood to be badly allocated, and GPT-3's 1.7 tokens per parameter became the canonical example of an under-trained model — the comparison rung 5 uses to make its case.

The paper is unusually candid about limits. GPT-3 makes things up — fluent, confident text that is simply wrong. It loses the thread over long passages, contradicting itself. It stumbles on tasks needing real multi-step reasoning, and its arithmetic falls apart past a few digits. Because it learned from the open internet, it also absorbed the web's biases around gender, race, and religion.

There's a deeper gap too: GPT-3 is trained only to continue text, not to follow instructions or be helpful. Ask it a question and it might answer — or it might continue with more questions, because that's a plausible continuation. Aligning that raw next-word engine to human intent was the unfinished business that InstructGPT and RLHF picked up next, and which ultimately produced ChatGPT.

GPT-3 is the hinge between two eras. Before it, you adapted models by training them; after it, you adapted them by talking to them. In-context learning turned the prompt into the primary interface for AI, and made "few-shot" and "zero-shot" everyday vocabulary. Every prompt you write to a modern assistant is a descendant of this paper.

It also settled, loudly, that scale delivers — the conviction that drove the models that followed. GPT-3 wasn't aligned, wasn't a chatbot, and got plenty wrong. But it proved that a big enough next-word predictor is a general-purpose, promptable engine, and everything from InstructGPT to today's frontier models is a refinement of what GPT-3 first made undeniable.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →