BERT,
explained.

This is rung 1 of the pre-training ladder, so there is no rung below it to measure against — the comparison is against the practice it replaced. Before BERT, the strong recipe for a language representation was to read a sentence left to right and predict the next word, because that is what a language model is. Every word's representation could therefore only see the words before it.

BERT's change is small to describe and large in consequence: stop predicting the next word, and instead hide about 15% of the tokens and predict those. Once the target is a hidden token rather than the next one, there is no reason to withhold the right-hand context, so every token's representation is conditioned on the whole sentence in both directions at once.

The units to carry up the ladder: 110M parameters for the base model and 340M for the large one — the largest a language-understanding model had usefully been at the time, and a figure worth remembering when rung 3 arrives at 175B. Note also what BERT is for: it produces representations that you then fine-tune, with a task-specific head, on labelled data for each task. Both halves of that sentence are things later rungs remove.

The Transformer had already shown that attention could replace recurrence. But the language models built on it still read like someone with a piece of paper covering the rest of the page: strictly left to right, one word at a time, predicting the next word from everything before it.

That one-way habit throws away half the clues. Take the word "bank" in "I went to the bank to deposit my" — you can't be sure it's a riverbank or a money bank until you read the word that comes after it. A left-to-right model, by design, never gets to peek forward. The meaning it builds is always half-blind.

BERT's bet was simple to state and awkward to pull off: let every word see the words on both sides of it, simultaneously. That's what the "B" stands for — Bidirectional Encoder Representations from Transformers.

The catch is a chicken-and-egg problem. If you train a model to predict the next word while also letting it look ahead, it can just read the answer — the future word it's supposed to guess is sitting right there. True bidirectional context and next-word prediction can't coexist. So BERT threw out next-word prediction entirely and invented a different game.

Instead of predicting the next word, BERT plays fill-in-the-blank. Take a sentence, randomly hide about 15% of its words behind a [MASK] token, and train the model to recover what was hidden — using every other word, on both sides, as evidence. This is masked language modeling (MLM).

Because the answer is now hidden rather than sitting one step ahead, the model is free to use the future as well as the past. To guess "mat," it leans on "cat," "sat," and "on" all at once. Do this across billions of words and the model is forced to learn deep, two-sided representations of meaning.

Filling in words teaches BERT about individual sentences. But many real tasks — question answering, natural-language inference — hinge on the relationship between two sentences. So BERT added a second pre-training task: next-sentence prediction (NSP). Show the model two sentences and ask a yes/no question — did B actually follow A in the original text, or is it a random impostor pulled from somewhere else?

Half the training pairs are genuine neighbors, half are random. Learning to tell them apart nudges BERT toward understanding discourse and coherence, not just word-level patterns. (Later work — RoBERTa — showed NSP was largely unnecessary, but it was part of the original recipe.)

BERT wraps its input in a few special tokens. Every sequence starts with [CLS] — a slot whose final vector becomes a summary of the whole input, perfect to hang a classifier on. Two sentences are divided by a [SEP] separator, and each token also carries a segment embedding marking whether it belongs to sentence A or B.

Words themselves are split into WordPiece sub-word units, so rare words like "tokenization" become "token" + "##ization" — a fixed 30,000-token vocabulary that never hits an unknown word. Each token's final embedding is the sum of three signals: what the word is, which segment it's in, and where it sits.

This is BERT's real gift to the field. Pre-training — the expensive part — happens once: MLM plus NSP over BooksCorpus and English Wikipedia, days of compute on a big cluster. The result is a model that already understands language in general.

Then, for any specific task, you fine-tune: bolt a tiny task-specific layer onto the [CLS] output (or every token, for tagging), and train briefly on a small labeled dataset. Sentiment, question answering, named-entity recognition — same pre-trained body, different lightweight head, a few hours each. Teams that couldn't dream of training a language model from scratch could now reach state-of-the-art by fine-tuning BERT.

The two model sizes aren't arbitrary — they fall straight out of the architecture with the same napkin formula that sizes every Transformer (≈12·d² weights per layer, plus the embedding matrix):

(30,522 is the WordPiece vocabulary; 512 the max positions.) The masking recipe is just as concrete. For every training sequence, 15% of tokens are selected for prediction — but only 80% of those become [MASK]; 10% are swapped for a random word and 10% left untouched:

That 80/10/10 hedge looks fussy until you see the failure it prevents: a model trained only on literal [MASK] slots learns "the interesting positions are the masked ones" — and at fine-tune time there are no masks, so the signal vanishes. The hedge forces every position to carry a real representation.

The paper is candid that masked language modelling creates a mismatch between pre-training and fine-tuning: the [MASK] token appears constantly during pre-training and never at inference. That is the whole reason for the 80/10/10 split the exercise above implements — of the tokens selected for prediction, most are masked, some are replaced with a random token, and some are left alone. It is a patch on a known defect, not an elegance.

The authors also note the obvious cost: because only the selected fraction of tokens is predicted, each batch teaches the model less than left-to-right prediction would, and pre-training converges more slowly per token seen.

How it held up: the bidirectional-encoder recipe was the dominant approach to language understanding for roughly three years and remains the right tool for retrieval and classification — rung 3 of the retrieval ladder is a direct descendant. What did not hold up is the assumption underneath it, that you need a task-specific fine-tune per task at all. Rung 2 makes every task the same shape, and rung 3 removes the fine-tune. Later work also showed BERT-sized models were badly under-trained on data relative to their parameter count, which is exactly the axis rung 5 formalises.

BERT didn't just edge ahead — it swept. On the GLUE benchmark of nine language-understanding tasks, BERT-large pushed the score to 80.5%, a 7.7-point jump over the previous best. On SQuAD v1.1 question answering it reached 93.2 F1 — past the 91.2 human baseline — and it set new records on eleven tasks at once.

The two sizes tell their own story: BERT-large, with 24 Transformer-encoder layers and 340M parameters, beat BERT-base everywhere — an early, clear signal that with the right pre-training, bigger keeps helping.

BERT is the encoder half of the family tree. Where GPT took the Transformer's decoder and ran toward generation, BERT took the encoder and ran toward understanding — and for years, if you were classifying, retrieving, or extracting, you reached for BERT or one of its descendants (RoBERTa, DistilBERT, ELECTRA, DeBERTa). Its sentence embeddings still power a huge share of search and retrieval systems today.

More than any single benchmark, BERT's legacy is the recipe it normalized: pre-train a Transformer on oceans of unlabeled text, then fine-tune. That idea — not the specific masking trick — is what every model after it inherited.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →