T5,
explained.

BERT (rung 1) gives you a representation and then a different head, a different loss and a different output format for each task — classification, span extraction, regression. T5's change is to delete that variety: every task is text in, text out. Translation, summarisation, sentiment, even a regression target are all expressed as a string to be generated. One model, one loss, one interface.

Stated in rung 1's units: BERT's largest useful configuration was 340M parameters; T5's largest is 11B, a ~32× jump. And where BERT was pre-trained on a curated corpus of books and Wikipedia, T5 introduces a filtered web-scale corpus (C4), which is the first appearance on this ladder of the idea that the dataset is a designed artefact rather than a given.

The other reason this rung matters for what follows: the paper is explicitly an empirical survey. It holds the text-to-text interface fixed and ablates the rest — architecture, objective, corpus, training length — to find which choices actually pay. That methodology is the ancestor of rungs 4 and 5, which do the same thing to a single question.

Post-BERT, transfer learning had swept NLP: pre-train once, fine-tune per task. But "fine-tune" hid a mess. Classification bolted a softmax head on the [CLS] token. Span-extraction tasks predicted start/end positions. Sequence tagging labeled every token. Translation used a separate encoder-decoder entirely. Each task had its own output layer, its own loss, its own plumbing.

That fragmentation made it hard to compare ideas cleanly — was a gain due to the model, the objective, the data, or the task-specific head? The T5 authors wanted one framework general enough to express every task the same way, so they could study what really drove transfer learning without the confound of a dozen architectures.

T5's move is to treat every NLP problem as mapping an input string to an output string. No task-specific heads. The label isn't a class id — it's the literal word. Sentiment analysis outputs the text "positive." Translation outputs the translated sentence. Even regression-style tasks output a number rendered as text.

A short task prefix in the input tells the model which job to do. Prepend "translate English to German:" and the same model translates; prepend "summarize:" and it summarizes. One model, one training objective, one decoding process — the task lives in the words, not the architecture.

Uniformity buys leverage. Because every task shares the same input/output shape and loss (predict the next token of the target text), a single set of hyperparameters, a single fine-tuning procedure, and a single model can cover them all. You can even train on many tasks at once — multi-task learning — simply by mixing their examples, since they're already the same format.

It also makes the model a clean experimental testbed. Want to know if a different pre-training objective helps? Swap it and re-measure across every task with no architecture changes. That controlled comparability is exactly what the paper was built to exploit.

Where BERT kept only the Transformer's encoder and GPT kept only the decoder, T5 uses the full encoder-decoder — the original Transformer shape. The encoder reads the input text bidirectionally; the decoder generates the output text one token at a time, attending to the encoder. Because the output is always generated text, this design fits the text-to-text framing naturally.

The paper's careful comparison found that for this unified, generation-based setup, the encoder-decoder generally beat encoder-only or decoder-only alternatives — a reminder that the "best" architecture depends on how you frame the problem, not an absolute ranking.

T5's pre-training task is span corruption. Take a sentence, drop out random contiguous spans of words, replace each with a unique sentinel token, and train the model to generate the missing spans as its target text. It's masked language modeling reimagined for text-to-text: instead of predicting single hidden tokens in place, the model produces the removed pieces as an output sequence. Concretely, with two spans dropped:

The sentinels line up input to target, so the pair is perfectly invertible — the original is fully reconstructable.

Feeding this needs data — enormous, clean data. So the team built C4, the Colossal Clean Crawled Corpus: ~750GB of English web text scraped from Common Crawl and aggressively filtered to strip boilerplate, duplicates, offensive text, and non-English pages. C4 outlived the paper, becoming a standard pre-training corpus across the field.

T5 is as much a survey as a model. Using the unified framework as a control, the authors ran a huge, methodical sweep: which architecture, which pre-training objective, which dataset, how much data, which fine-tuning and multi-task strategy, and how it all scales. Because every variant shared the same text-to-text harness, the comparisons were unusually clean.

The flagship T5 model (11 billion parameters) set state-of-the-art on many benchmarks, but the durable value is the recipe and the ablations — a well-lit map of which knobs matter for transfer learning, backed by a single consistent framework.

Text-to-text is elegant but not free. Emitting labels as text is slower than a classification head reading off one logit, and generation can produce outputs outside the valid label set that you must constrain or post-process. The encoder-decoder also has roughly double the parameters of a decoder-only model at the same depth.

And T5 still fine-tunes per task — it predates the in-context, prompt-only paradigm that GPT-3 popularized the same year. It's the peak of the "pre-train then fine-tune" era rather than a step beyond it, and it inherits that era's hunger for labeled data at fine-tuning time.

The text-to-text idea won. Framing tasks as strings — with an instruction in the input and an answer in the output — is exactly how we prompt modern chat models. When you type "summarize this" or "translate to French" to an assistant, you're using T5's mental model, now scaled up and made zero-shot.

T5's descendants (mT5 for multilingual, Flan-T5 for instruction tuning, and others) remain workhorses for encoder-decoder tasks like summarization and translation. And its greatest gift may be methodological: it showed the value of a unified framework plus rigorous ablation, a template for how to study these systems honestly rather than chase one benchmark.

The paper is unusually direct that its results are relative, not absolute: it compares choices under a fixed budget, and it warns that the winning configuration at one scale need not win at another. Read after rung 5, that caveat is close to prophetic — several of the field's later corrections are exactly the case of a conclusion drawn at one scale failing at a larger one.

It also concedes that the encoder-decoder shape it lands on costs roughly twice the parameters of a decoder-only model for the same layer count, and justifies it on measured quality rather than principle.

How it held up: the text-to-text interface won completely — every instruction-following model on rungs above this one is text in, text out, and the "prompt" is that interface under a different name. The encoder-decoder architecture did not win; the field went decoder-only from rung 3 onward, largely for generation efficiency. The survey methodology won twice over, and the C4-style filtered web corpus became the default base of every pre-training mix until rung 6 questions filtering itself.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →