GPT-3,
explained.

BERT had made a recipe famous: pre-train a big model once, then fine-tune it on each specific task with a labeled dataset. It worked brilliantly — but it had a cost that was easy to overlook. Every new task needed its own curated collection of thousands of labeled examples, and its own round of training that nudged the model's weights.

Humans don't work that way. You can be told "translate these to French" and shown one or two examples, and you're off — no dataset of ten thousand pairs, no retraining of your brain. The GPT-3 authors framed this squarely: the fine-tuning recipe was powerful but narrow, and it scaled poorly to the long tail of tasks nobody had built a dataset for. They wanted a model that could pick up a task the way a person does — from a short description and a handful of examples.

GPT-3's architecture is almost boringly familiar: it's the same autoregressive Transformer decoder as GPT-2, trained on the same humble objective — predict the next word, over and over, across a huge slice of the internet. The paper introduces essentially no new architecture. Its one radical move is scale.

GPT-2 had 1.5 billion parameters. GPT-3 has 175 billion — over a hundred times larger — trained on roughly 300 billion tokens of text. The hypothesis was that capability wasn't waiting on a cleverer design but on size, and that at some scale, new behaviors would simply emerge. The title says it all: Language Models are Few-Shot Learners. Scale was the whole idea.

The paper's central move is to stop fine-tuning entirely and instead put the "training" inside the prompt. It measures three settings, all with the model's weights frozen:

Zero-shot — you give only a task description. One-shot — you add a single worked example. Few-shot — you add several (often 10–100) examples before the real question. In every case the model just reads the prompt and continues it.

Nothing here updates a single weight. The examples are just text the model reads on its way to generating the answer. And yet, at 175B, accuracy climbs steadily as you go from zero- to one- to few-shot — the extra examples genuinely help. Smaller models barely benefit; the effect grows with scale.

GPT-3 is a stack of 96 Transformer-decoder layers, 12,288-dimensional embeddings, 96 attention heads, and a context window of 2,048 tokens. Each layer is the standard decoder block: masked self-attention (each token attends only to tokens before it) followed by a feed-forward network. Trained on next-token prediction, it becomes a probability machine — given everything so far, what word comes next?

That single objective is deceptively powerful. To predict the next word well across the whole internet, the model has to implicitly pick up grammar, facts, translation, arithmetic, and reasoning patterns — because all of those show up in text. Few-shot prompting simply conditions that machine: the examples in the prompt shift the probabilities so the most likely continuation is the answer in the format you demonstrated.

The headline phenomenon has a name: in-context learning. The model appears to "learn" the task from the examples in its context window, during a single forward pass, with no gradient descent and no weight change anywhere. Whatever adaptation is happening lives entirely in the activations as the model processes the prompt.

This was genuinely unexpected. Learning, in machine learning, had always meant updating weights. GPT-3 showed a second, temporary kind of learning that happens at inference time, purely from what's in front of the model. It reframed the interface to AI: you no longer train a model for a task, you describe the task and show examples. That's the birth of prompting as a discipline.

Across dozens of benchmarks, few-shot GPT-3 was often competitive with — sometimes matching — fine-tuned models that had seen thousands of task-specific examples. It did strong translation with no parallel-text fine-tuning, answered trivia, unscrambled words, and even did multi-digit arithmetic it was never explicitly taught. The clearest pattern was that more examples and more parameters both helped.

The scaling curves were the real message. Plot accuracy against model size and it keeps climbing — bigger models were not just a little better, they were qualitatively better at learning from context. The gap between few-shot and zero-shot also widened with scale, evidence that in-context learning is something large models grow into.

The paper is unusually candid about limits. GPT-3 makes things up — fluent, confident text that is simply wrong. It loses the thread over long passages, contradicting itself. It stumbles on tasks needing real multi-step reasoning, and its arithmetic falls apart past a few digits. Because it learned from the open internet, it also absorbed the web's biases around gender, race, and religion.

There's a deeper gap too: GPT-3 is trained only to continue text, not to follow instructions or be helpful. Ask it a question and it might answer — or it might continue with more questions, because that's a plausible continuation. Aligning that raw next-word engine to human intent was the unfinished business that InstructGPT and RLHF picked up next, and which ultimately produced ChatGPT.

GPT-3 is the hinge between two eras. Before it, you adapted models by training them; after it, you adapted them by talking to them. In-context learning turned the prompt into the primary interface for AI, and made "few-shot" and "zero-shot" everyday vocabulary. Every prompt you write to a modern assistant is a descendant of this paper.

It also settled, loudly, that scale delivers — the conviction that drove the models that followed. GPT-3 wasn't aligned, wasn't a chatbot, and got plenty wrong. But it proved that a big enough next-word predictor is a general-purpose, promptable engine, and everything from InstructGPT to today's frontier models is a refinement of what GPT-3 first made undeniable.