BERT,
explained.

This is rung 1 of the pre-training ladder, so there is no rung below it to measure against — the comparison is against the practice it replaced. Before BERT, the strong recipe for a language representation was to read a sentence left to right and predict the next word, because that is what a language model is. Every word's representation could therefore only see the words before it.

BERT's change is small to describe and large in consequence: stop predicting the next word, and instead hide about 15% of the tokens and predict those. Once the target is a hidden token rather than the next one, there is no reason to withhold the right-hand context, so every token's representation is conditioned on the whole sentence in both directions at once.

The units to carry up the ladder: 110M parameters for the base model and 340M for the large one — the largest a language-understanding model had usefully been at the time, and a figure worth remembering when rung 3 arrives at 175B. Note also what BERT is for: it produces representations that you then fine-tune, with a task-specific head, on labelled data for each task. Both halves of that sentence are things later rungs remove.

The Transformer had already shown that attention could replace recurrence. But the language models built on it still read like someone with a piece of paper covering the rest of the page: strictly left to right, one word at a time, predicting the next word from everything before it.

That one-way habit throws away half the clues. Take the word "bank" in "I went to the bank to deposit my" — you can't be sure it's a riverbank or a money bank until you read the word that comes after it. A left-to-right model, by design, never gets to peek forward. The meaning it builds is always half-blind.

BERT's bet was simple to state and awkward to pull off: let every word see the words on both sides of it, simultaneously. That's what the "B" stands for — Bidirectional Encoder Representations from Transformers.

The catch is a chicken-and-egg problem. If you train a model to predict the next word while also letting it look ahead, it can just read the answer — the future word it's supposed to guess is sitting right there. True bidirectional context and next-word prediction can't coexist. So BERT threw out next-word prediction entirely and invented a different game.

Instead of predicting the next word, BERT plays fill-in-the-blank. Take a sentence, randomly hide about 15% of its words behind a [MASK] token, and train the model to recover what was hidden — using every other word, on both sides, as evidence. This is masked language modeling (MLM).

Because the answer is now hidden rather than sitting one step ahead, the model is free to use the future as well as the past. To guess "mat," it leans on "cat," "sat," and "on" all at once. Do this across billions of words and the model is forced to learn deep, two-sided representations of meaning.

Filling in words teaches BERT about individual sentences. But many real tasks — question answering, natural-language inference — hinge on the relationship between two sentences. So BERT added a second pre-training task: next-sentence prediction (NSP). Show the model two sentences and ask a yes/no question — did B actually follow A in the original text, or is it a random impostor pulled from somewhere else?

Half the training pairs are genuine neighbors, half are random. Learning to tell them apart nudges BERT toward understanding discourse and coherence, not just word-level patterns. (Later work — RoBERTa — showed NSP was largely unnecessary, but it was part of the original recipe.)

BERT wraps its input in a few special tokens. Every sequence starts with [CLS] — a slot whose final vector becomes a summary of the whole input, perfect to hang a classifier on. Two sentences are divided by a [SEP] separator, and each token also carries a segment embedding marking whether it belongs to sentence A or B.

Words themselves are split into WordPiece sub-word units, so rare words like "tokenization" become "token" + "##ization" — a fixed 30,000-token vocabulary that never hits an unknown word. Each token's final embedding is the sum of three signals: what the word is, which segment it's in, and where it sits.

This is BERT's real gift to the field. Pre-training — the expensive part — happens once: MLM plus NSP over BooksCorpus and English Wikipedia, days of compute on a big cluster. The result is a model that already understands language in general.

Then, for any specific task, you fine-tune: bolt a tiny task-specific layer onto the [CLS] output (or every token, for tagging), and train briefly on a small labeled dataset. Sentiment, question answering, named-entity recognition — same pre-trained body, different lightweight head, a few hours each. Teams that couldn't dream of training a language model from scratch could now reach state-of-the-art by fine-tuning BERT.

The two model sizes aren't arbitrary — they fall straight out of the architecture with the same napkin formula that sizes every Transformer (≈12·d² weights per layer, plus the embedding matrix):

(30,522 is the WordPiece vocabulary; 512 the max positions.) The masking recipe is just as concrete. For every training sequence, 15% of tokens are selected for prediction — but only 80% of those become [MASK]; 10% are swapped for a random word and 10% left untouched:

That 80/10/10 hedge looks fussy until you see the failure it prevents: a model trained only on literal [MASK] slots learns "the interesting positions are the masked ones" — and at fine-tune time there are no masks, so the signal vanishes. The hedge forces every position to carry a real representation.

BERT's method has two phases that share one network. Pre-training runs once, on unlabeled text, and is the expensive part. Fine-tuning runs once per task, on labeled data, and is cheap. This is the procedure the paper describes.

1. Build a document-level corpus. BooksCorpus (about 800M words) plus English Wikipedia (about 2,500M words), roughly 3.3B words in all. From Wikipedia only text passages are kept; lists, tables and headers are dropped. The authors chose whole documents over a shuffled sentence-level corpus such as the Billion Word Benchmark, because the model is trained on long contiguous spans.
2. Sample a pair and pack it. Each example is two spans of text, A and B, with a combined length of at most 512 tokens. Half the time B is the span that really follows A; half the time it is a random span from elsewhere in the corpus. The pair is packed as [CLS] A [SEP] B [SEP], and segment embeddings mark which tokens belong to which span.
3. Corrupt 15% of positions. WordPiece positions are chosen at random and each chosen one is replaced by [MASK], by a random token, or left alone, in the 80/10/10 proportions. The network only ever sees the corrupted sequence.
4. Encode. A Transformer encoder with no causal mask. BERT-BASE has 12 layers, hidden size 768 and 12 attention heads; BERT-LARGE has 24 layers, hidden size 1,024 and 16 heads. The feed-forward inner size is four times the hidden size, and the activation is GELU.
5. Score two losses. Final vectors at the chosen positions go through a softmax over the whole vocabulary and are scored by cross-entropy against the original tokens; every other position contributes nothing. The final [CLS] vector feeds a binary IsNext / NotNext classifier. The training loss is the sum of the two mean losses.
6. Optimize. Adam with learning rate 1e-4, β₁ = 0.9, β₂ = 0.999 and L2 weight decay 0.01; linear warmup over the first 10,000 steps, then linear decay; dropout 0.1 on every layer. Each batch is 256 sequences of up to 512 tokens, about 128,000 tokens, for 1,000,000 steps, which the paper puts at roughly 40 epochs over the corpus. Attention cost grows with the square of length, so 90% of the steps use 128-token sequences and only the last 10% use 512, mainly so the later position embeddings get trained.
7. Use modest hardware by later standards. BERT-BASE trained on 16 TPU chips and BERT-LARGE on 64, about four days each.
8. Fine-tune end to end. Add a task head and train every weight, body included, on the labeled data.

The three kinds of head

For classification, one weight matrix W, sized labels × hidden, reads the final [CLS] vector C, and the label probabilities are softmax(C·W). For tagging, as in named-entity recognition, a classifier reads every token's final vector; the paper uses the first sub-token of each word. For SQuAD, the question is segment A, the passage is segment B, and the only new parameters are a start vector S and an end vector E.

The suggested search is small: batch size 16 or 32, learning rate 5e-5, 3e-5 or 2e-5, and 2 to 4 epochs. The paper notes that every result it reports can be reproduced in at most an hour on one Cloud TPU, or a few hours on a GPU, from the same checkpoint. BERT-LARGE sometimes failed to fine-tune on small datasets, so the authors ran several random restarts and kept the best model on the dev set.

• M — the positions chosen for prediction, about 15% of the sequence; x̃ is the corrupted input and xi the original token at position i.
• p(xi | x̃) — a softmax over the WordPiece vocabulary computed from the final vector at position i, which has seen the whole corrupted sequence in both directions.
• LNSP — binary cross-entropy of the IsNext classifier on the final [CLS] vector.
• Ti — final vector of passage token i; S and E — the only new SQuAD parameters, each the size of a hidden state. Training maximizes log P_start of the true start plus log P_end of the true end.

At the extremes: if M covered every position there would be nothing left to condition on, and if M were empty there would be no signal, which is why a modest fraction is chosen. The constraint j ≥ i rules out an end before its start. For SQuAD v2.0 the paper scores "no answer" as a span at [CLS], s_null = S·C + E·C, and answers only when the best span beats s_null + τ, with τ tuned on the dev set.

Each model was fine-tuned separately on each task. BERT-BASE was sized to match OpenAI GPT, a left-to-right Transformer, so the two could be compared directly; BERT-LARGE shows what size adds. BERT set a new state of the art on eleven tasks.

BERT-LARGE beat BERT-BASE on every GLUE task, most clearly on the small datasets.

Ablations, all on BERT-BASE dev sets

Dropping next-sentence prediction hurt QNLI, MNLI and SQuAD, with QNLI falling from 88.4 to 84.9. Training left to right without it hurt far more: MRPC fell from 86.7 to 77.5 and SQuAD F1 from 88.5 to 77.8. Adding a randomly initialized BiLSTM on top of that left-to-right model recovered SQuAD partway, to 84.9, but made GLUE worse. Scaling from 3 layers up to BERT-LARGE improved every task tried, even MRPC with only a few thousand training examples, and training for 1M rather than 500k steps added about a point on MNLI.

A feature-based variant, with BERT frozen and its top four hidden layers concatenated as input to a BiLSTM tagger on CoNLL-2003 named entities, reached 96.1 dev F1, only 0.3 behind fine-tuning the whole model. The masking-mix ablation found fine-tuning robust to the 80/10/10 choice, while the feature-based setting suffered most when every chosen token became [MASK].

The paper is candid that masked language modelling creates a mismatch between pre-training and fine-tuning: the [MASK] token appears constantly during pre-training and never at inference. That is the whole reason for the 80/10/10 split the exercise above implements — of the tokens selected for prediction, most are masked, some are replaced with a random token, and some are left alone. It is a patch on a known defect, not an elegance.

The authors also note the obvious cost: because only the selected fraction of tokens is predicted, each batch teaches the model less than left-to-right prediction would, and pre-training converges more slowly per token seen.

How it held up: the bidirectional-encoder recipe was the dominant approach to language understanding for roughly three years and remains the right tool for retrieval and classification — rung 3 of the retrieval ladder is a direct descendant. What did not hold up is the assumption underneath it, that you need a task-specific fine-tune per task at all. Rung 2 makes every task the same shape, and rung 3 removes the fine-tune. Later work also showed BERT-sized models were badly under-trained on data relative to their parameter count, which is exactly the axis rung 5 formalises.

BERT is the encoder half of the family tree. Where GPT took the Transformer's decoder and ran toward generation, BERT took the encoder and ran toward understanding — and for years, if you were classifying, retrieving, or extracting, you reached for BERT or one of its descendants (RoBERTa, DistilBERT, ELECTRA, DeBERTa). Its sentence embeddings still power a huge share of search and retrieval systems today.

More than any single benchmark, BERT's legacy is the recipe it normalized: pre-train a Transformer on oceans of unlabeled text, then fine-tune. That idea — not the specific masking trick — is what every model after it inherited.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The ladder where rung 5 overturns rung 4. Read `scaling-laws` before `chinchilla` — the second measures the same axis in the same units and gets a different answer.

1. 1BERT2019
2. 2T52020
3. 3GPT-32020
4. 4Scaling Laws2020
5. 5Chinchilla2022
6. 6Textbooks Are All You Need2023
7. 7Muon2024

All 110 papers, in 19 ladders →