GPT-3,
explained.

Rungs 1 and 2 both end with a gradient step on your task. BERT fine-tunes a head; T5 fine-tunes the whole model on a task expressed as text. GPT-3's claim is that at sufficient scale you can skip that entirely: put a few worked examples in the prompt, and the model performs the task without any weight update at all. No labelled training set, no optimiser, no checkpoint per task.

In the ladder's units: T5's largest was 11B parameters; GPT-3 is 175B, a ~16× jump, trained on roughly 300B tokens. Hold those two numbers together, because rungs 4 and 5 are an argument about precisely that ratio: 175B parameters against 300B tokens is about 1.7 training tokens per parameter. Nothing in this paper suggests that ratio is a choice at all — it reads as a consequence of using the compute available.

The interface change is the lasting one. Once the way to get behaviour out of a model is to write text at it, prompting becomes an engineering discipline, and everything on the reasoning ladder becomes possible.

BERT had made a recipe famous: pre-train a big model once, then fine-tune it on each specific task with a labeled dataset. It worked brilliantly — but it had a cost that was easy to overlook. Every new task needed its own curated collection of thousands of labeled examples, and its own round of training that nudged the model's weights.

Humans don't work that way. You can be told "translate these to French" and shown one or two examples, and you're off — no dataset of ten thousand pairs, no retraining of your brain. The GPT-3 authors framed this squarely: the fine-tuning recipe was powerful but narrow, and it scaled poorly to the long tail of tasks nobody had built a dataset for. They wanted a model that could pick up a task the way a person does — from a short description and a handful of examples.

GPT-3's architecture is almost boringly familiar: it's the same autoregressive Transformer decoder as GPT-2, trained on the same humble objective — predict the next word, over and over, across a huge slice of the internet. The paper introduces essentially no new architecture. Its one radical move is scale.

GPT-2 had 1.5 billion parameters. GPT-3 has 175 billion — over a hundred times larger — trained on roughly 300 billion tokens of text. The hypothesis was that capability wasn't waiting on a cleverer design but on size, and that at some scale, new behaviors would simply emerge. The title says it all: Language Models are Few-Shot Learners. Scale was the whole idea.

The paper's central move is to stop fine-tuning entirely and instead put the "training" inside the prompt. It measures three settings, all with the model's weights frozen:

Zero-shot — you give only a task description. One-shot — you add a single worked example. Few-shot — you add several (often 10–100) examples before the real question. In every case the model just reads the prompt and continues it.

Nothing here updates a single weight. The examples are just text the model reads on its way to generating the answer. And yet, at 175B, accuracy climbs steadily as you go from zero- to one- to few-shot — the extra examples genuinely help. Smaller models barely benefit; the effect grows with scale.

GPT-3 is a stack of 96 Transformer-decoder layers, 12,288-dimensional embeddings, 96 attention heads, and a context window of 2,048 tokens. Each layer is the standard decoder block: masked self-attention (each token attends only to tokens before it) followed by a feed-forward network. Trained on next-token prediction, it becomes a probability machine — given everything so far, what word comes next?

That single objective is deceptively powerful. To predict the next word well across the whole internet, the model has to implicitly pick up grammar, facts, translation, arithmetic, and reasoning patterns — because all of those show up in text. Few-shot prompting simply conditions that machine: the examples in the prompt shift the probabilities so the most likely continuation is the answer in the format you demonstrated.

"175 billion parameters" sounds like a press-release number, but you can compute it from the architecture on a napkin. A Transformer layer's weights are dominated by four attention projections (each d×d) and two feed-forward matrices (d×4d and 4d×d), giving ≈ 12·d² parameters per layer:

The same formula reproduces the whole model family the paper trained (embedding matrix ≈ 50257·d added; it matters for the small ones, vanishes at scale):

Training cost falls out just as cleanly. Each token requires roughly 6 FLOPs per parameter (2 for the forward pass, 4 for backward), and GPT-3 saw 300B tokens:

That's several thousand petaflop/s-days. These two rules — 12·L·d² and 6·N·D — are the napkin math practitioners still use to size and price every model since.

The headline phenomenon has a name: in-context learning. The model appears to "learn" the task from the examples in its context window, during a single forward pass, with no gradient descent and no weight change anywhere. Whatever adaptation is happening lives entirely in the activations as the model processes the prompt.

This was genuinely unexpected. Learning, in machine learning, had always meant updating weights. GPT-3 showed a second, temporary kind of learning that happens at inference time, purely from what's in front of the model. It reframed the interface to AI: you no longer train a model for a task, you describe the task and show examples. That's the birth of prompting as a discipline.

The paper's own limitations section is longer and franker than most: few-shot performance is still well short of fine-tuned state-of-the-art on many tasks; the model is weak on tasks requiring comparison of two sentences; it has no notion of which of its outputs are true; and the authors flag both the training-data contamination problem and the energy cost of the run. They also state plainly that the model reproduces the biases of its corpus.

The admission most relevant to this ladder is quieter: they note the model is trained once, and that they could not afford to retrain it to explore alternatives. A single un-repeatable run is not a measurement of how to spend compute — which is the gap rung 4 sets out to fill.

How it held up: in-context learning held up completely and is now the primary interface to every frontier model. The specific sizing did not. Within two years the same compute was understood to be badly allocated, and GPT-3's 1.7 tokens per parameter became the canonical example of an under-trained model — the comparison rung 5 uses to make its case.

Across dozens of benchmarks, few-shot GPT-3 was often competitive with — sometimes matching — fine-tuned models that had seen thousands of task-specific examples. It did strong translation with no parallel-text fine-tuning, answered trivia, unscrambled words, and even did multi-digit arithmetic it was never explicitly taught. The clearest pattern was that more examples and more parameters both helped.

The paper's own headline numbers, not paraphrases:

The scaling curves were the real message. Plot accuracy against model size and it keeps climbing — bigger models were not just a little better, they were qualitatively better at learning from context. The gap between few-shot and zero-shot also widened with scale, evidence that in-context learning is something large models grow into.

The paper is unusually candid about limits. GPT-3 makes things up — fluent, confident text that is simply wrong. It loses the thread over long passages, contradicting itself. It stumbles on tasks needing real multi-step reasoning, and its arithmetic falls apart past a few digits. Because it learned from the open internet, it also absorbed the web's biases around gender, race, and religion.

There's a deeper gap too: GPT-3 is trained only to continue text, not to follow instructions or be helpful. Ask it a question and it might answer — or it might continue with more questions, because that's a plausible continuation. Aligning that raw next-word engine to human intent was the unfinished business that InstructGPT and RLHF picked up next, and which ultimately produced ChatGPT.

GPT-3 is the hinge between two eras. Before it, you adapted models by training them; after it, you adapted them by talking to them. In-context learning turned the prompt into the primary interface for AI, and made "few-shot" and "zero-shot" everyday vocabulary. Every prompt you write to a modern assistant is a descendant of this paper.

It also settled, loudly, that scale delivers — the conviction that drove the models that followed. GPT-3 wasn't aligned, wasn't a chatbot, and got plenty wrong. But it proved that a big enough next-word predictor is a general-purpose, promptable engine, and everything from InstructGPT to today's frontier models is a refinement of what GPT-3 first made undeniable.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →