BERT (Bidirectional Encoder Representations from Transformers) is a 2018 model from Google that pre-trains a Transformer encoder on huge amounts of unlabeled text, then fine-tunes it on specific tasks. Its key idea is reading text bidirectionally — using both left and right context at once — which made pre-training the default recipe for modern NLP.

What is the GLUE benchmark?

GLUE (General Language Understanding Evaluation) is a collection of nine English language-understanding tasks used to measure NLP models. BERT pushed the GLUE score to 80.5% on release — a 7.7-point jump over the previous best — which is a large part of why it made such an impact.

Paper Breakdowns / BERT

Paper 02~12 min readNAACL 2019worked math + runnable code

Paper Breakdown

BERT,
explained.

Q: What is masked language modeling?

Masked language modeling (MLM) is BERT's pre-training task: randomly hide about 15% of the words in a sentence and train the model to predict them from the surrounding context on both sides. Because the model must fill in a blank using words before and after it, it learns deeply bidirectional representations, unlike left-to-right language models.

Q: How is BERT different from GPT?

BERT is an encoder that reads the whole sentence bidirectionally and is best for understanding tasks (classification, question answering, tagging). GPT is a decoder that reads left-to-right and is built for generating text. BERT is fine-tuned per task; GPT-style models are increasingly used zero-shot with prompts.

Q: What does pre-train then fine-tune mean?

Pre-training teaches BERT general language from massive unlabeled text once, at great expense. Fine-tuning then adapts that pre-trained model to a specific task with a small labeled dataset and a little extra training. This transfer-learning recipe let teams reach state-of-the-art results without training a model from scratch.

One year after the Transformer, a Google team asked a stubborn question: why should a model only read a sentence left to right? BERT let it read both directions at once — and by teaching a model to fill in the blanks across a billion words, it made "pre-train once, fine-tune anywhere" the default recipe for modern NLP.

Video breakdown

The animated walkthrough is in production.

Read the full breakdown below in the meantime ↓

Prefer video? Browse published breakdowns →

The one-way problem Read both ways Masked language modeling Next-sentence prediction Tokens & inputs Pre-train, fine-tune The models, derived Run it yourself The results Why it still matters

The one-way problem

The Transformer had already shown that attention could replace recurrence. But the language models built on it still read like someone with a piece of paper covering the rest of the page: strictly left to right, one word at a time, predicting the next word from everything before it.

That one-way habit throws away half the clues. Take the word "bank" in "I went to the bank to deposit my" — you can't be sure it's a riverbank or a money bank until you read the word that comes after it. A left-to-right model, by design, never gets to peek forward. The meaning it builds is always half-blind.

The big idea: read both ways at once

BERT's bet was simple to state and awkward to pull off: let every word see the words on both sides of it, simultaneously. That's what the "B" stands for — Bidirectional Encoder Representations from Transformers.

The catch is a chicken-and-egg problem. If you train a model to predict the next word while also letting it look ahead, it can just read the answer — the future word it's supposed to guess is sitting right there. True bidirectional context and next-word prediction can't coexist. So BERT threw out next-word prediction entirely and invented a different game.

Masked language modeling: fill in the blank

Instead of predicting the next word, BERT plays fill-in-the-blank. Take a sentence, randomly hide about 15% of its words behind a [MASK] token, and train the model to recover what was hidden — using every other word, on both sides, as evidence. This is masked language modeling (MLM).

Masked language modeling

[CLS] the cat sat on the [MASK] → mat

Because the answer is now hidden rather than sitting one step ahead, the model is free to use the future as well as the past. To guess "mat," it leans on "cat," "sat," and "on" all at once. Do this across billions of words and the model is forced to learn deep, two-sided representations of meaning.

Worth knowing

A subtlety: the [MASK] token never appears at fine-tuning or inference time, which would create a mismatch. BERT hedges — of the 15% chosen words, 80% become [MASK], 10% are swapped for a random word, and 10% are left unchanged — so the model can't assume a masked slot is always literally [MASK].

Next-sentence prediction

Filling in words teaches BERT about individual sentences. But many real tasks — question answering, natural-language inference — hinge on the relationship between two sentences. So BERT added a second pre-training task: next-sentence prediction (NSP). Show the model two sentences and ask a yes/no question — did B actually follow A in the original text, or is it a random impostor pulled from somewhere else?

Half the training pairs are genuine neighbors, half are random. Learning to tell them apart nudges BERT toward understanding discourse and coherence, not just word-level patterns. (Later work — RoBERTa — showed NSP was largely unnecessary, but it was part of the original recipe.)

Tokens, [CLS], and [SEP]

BERT wraps its input in a few special tokens. Every sequence starts with [CLS] — a slot whose final vector becomes a summary of the whole input, perfect to hang a classifier on. Two sentences are divided by a [SEP] separator, and each token also carries a segment embedding marking whether it belongs to sentence A or B.

Words themselves are split into WordPiece sub-word units, so rare words like "tokenization" become "token" + "##ization" — a fixed 30,000-token vocabulary that never hits an unknown word. Each token's final embedding is the sum of three signals: what the word is, which segment it's in, and where it sits.

The recipe: pre-train once, fine-tune anywhere

This is BERT's real gift to the field. Pre-training — the expensive part — happens once: MLM plus NSP over BooksCorpus and English Wikipedia, days of compute on a big cluster. The result is a model that already understands language in general.

Then, for any specific task, you fine-tune: bolt a tiny task-specific layer onto the [CLS] output (or every token, for tagging), and train briefly on a small labeled dataset. Sentiment, question answering, named-entity recognition — same pre-trained body, different lightweight head, a few hours each. Teams that couldn't dream of training a language model from scratch could now reach state-of-the-art by fine-tuning BERT.

The shift

Before BERT, most NLP models were trained from scratch per task. After BERT, "download a pre-trained model and fine-tune it" became the default. That transfer-learning mindset is the through-line from BERT to every large model since.

110M and 340M, derived by hand

The two model sizes aren't arbitrary — they fall straight out of the architecture with the same napkin formula that sizes every Transformer (≈12·d² weights per layer, plus the embedding matrix):

BERT-base: 12·12·768² + 30522·768 + 512·768 ≈ 108.8M ≈ 110M ✓
BERT-large: 12·24·1024² + 30522·1024 + 512·1024 ≈ 333.8M ≈ 340M ✓

(30,522 is the WordPiece vocabulary; 512 the max positions.) The masking recipe is just as concrete. For every training sequence, 15% of tokens are selected for prediction — but only 80% of those become [MASK]; 10% are swapped for a random word and 10% left untouched:

Per 1,000 tokens	Count	Why
Selected for prediction (15%)	150	enough signal per pass without destroying context
→ replaced with [MASK] (80%)	120	the core fill-in-the-blank task
→ swapped for a random token (10%)	15	model can't trust that visible words are real — must model everything
→ left unchanged (10%)	15	[MASK] never appears at fine-tune time; this closes the mismatch

That 80/10/10 hedge looks fussy until you see the failure it prevents: a model trained only on literal [MASK] slots learns "the interesting positions are the masked ones" — and at fine-tune time there are no masks, so the signal vanishes. The hedge forces every position to carry a real representation.

RUN IT YOURSELF

The masking recipe & the napkin math

BERT's exact 15% / 80-10-10 masking procedure on real text, plus the parameter derivation for both model sizes — running in your browser through WebAssembly. Run it, check the proportions land where the paper says, then change MASK_RATE to 30% and think about what that does to the training signal (this exact question produced several follow-up papers).

CPython · WebAssembly

import random

MASK_RATE = 0.15          # fraction of tokens selected for prediction

def mlm_mask(tokens, seed=42):
    """BERT's masking recipe: 15% selected; of those 80% -> [MASK],
    10% -> random token, 10% left unchanged (but still predicted)."""
    rng = random.Random(seed)
    out = []
    for tok in tokens:
        if rng.random() >= MASK_RATE:
            out.append((tok, "keep-visible"))
            continue
        roll = rng.random()
        if roll < 0.8:
            out.append(("[MASK]", "predict"))
        elif roll < 0.9:
            out.append((rng.choice(tokens), "predict-random-swap"))
        else:
            out.append((tok, "predict-unchanged"))
    return out

sentence = "the quick brown fox jumps over the lazy dog near the river bank".split()
print("masked sentence:")
print("  ", " ".join(t for t, _ in mlm_mask(sentence)))

# proportions on a big sample
big = ["tok"] * 10000
kinds = {}
for _, kind in mlm_mask(big):
    kinds[kind] = kinds.get(kind, 0) + 1
print()
print("per 10,000 tokens:")
for k, v in sorted(kinds.items()):
    print(f"  {k:22} {v:5}  ({100*v/len(big):.1f}%)")

# the napkin math for the two model sizes
VOCAB, POSITIONS = 30522, 512

def bert_params(layers, d_model):
    return 12 * layers * d_model**2 + VOCAB * d_model + POSITIONS * d_model

print()
print(f"BERT-base : {bert_params(12, 768)/1e6:.1f}M  (paper: 110M)")
print(f"BERT-large: {bert_params(24, 1024)/1e6:.1f}M  (paper: 340M)")

# Try it: set MASK_RATE = 0.3 - later work found higher rates can be fine.

The real results

BERT didn't just edge ahead — it swept. On the GLUE benchmark of nine language-understanding tasks, BERT-large pushed the score to 80.5%, a 7.7-point jump over the previous best. On SQuAD v1.1 question answering it reached 93.2 F1 — past the 91.2 human baseline — and it set new records on eleven tasks at once.

Model	GLUE score	Note
Pre-BERT best (ELMo-era)	72.8	previous state of the art
OpenAI GPT (left-to-right)	75.1	one-directional
BERT-base (110M params)	79.6	12 layers
BERT-large (340M params)	80.5	24 layers

The two sizes tell their own story: BERT-large, with 24 Transformer-encoder layers and 340M parameters, beat BERT-base everywhere — an early, clear signal that with the right pre-training, bigger keeps helping.

Why it still matters

BERT is the encoder half of the family tree. Where GPT took the Transformer's decoder and ran toward generation, BERT took the encoder and ran toward understanding — and for years, if you were classifying, retrieving, or extracting, you reached for BERT or one of its descendants (RoBERTa, DistilBERT, ELECTRA, DeBERTa). Its sentence embeddings still power a huge share of search and retrieval systems today.

More than any single benchmark, BERT's legacy is the recipe it normalized: pre-train a Transformer on oceans of unlabeled text, then fine-tune. That idea — not the specific masking trick — is what every model after it inherited.

Worth knowing

When you build semantic search or a RAG retriever, the embedding model turning your text into vectors is very often a BERT descendant. Understanding BERT is understanding the retrieval side of modern AI.

Frequently asked

Quick answers

What is BERT?

A 2018 Google model that pre-trains a Transformer encoder on massive unlabeled text, then fine-tunes per task. Its signature is reading text bidirectionally — both left and right context at once.

What is masked language modeling?

Hide ~15% of a sentence's words and train the model to predict them from the words on both sides. That forces deeply bidirectional understanding, unlike left-to-right prediction.

How is BERT different from GPT?

BERT is an encoder built for understanding (classification, QA, tagging), read bidirectionally and fine-tuned per task. GPT is a decoder built for generation, read left-to-right.

What does pre-train then fine-tune mean?

Learn general language once from huge unlabeled data (expensive), then cheaply adapt that model to each specific task with a small labeled dataset and a lightweight head.

Why does BERT still matter?

Its descendants power much of today's search, retrieval, and embedding — including RAG retrievers — and it normalized the pre-train-then-fine-tune recipe used by every large model since.

Finished this one? 0 / 101 Paper Breakdowns done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering

More Paper Breakdowns

Cite this page

Reference it in your work, paper or an AI's context window.

APASingh, S. (2026). BERT. Vibe Engines. https://vibeengines.com/paper/bert

MLASingh, Saurabh. “BERT.” Vibe Engines, 2026, vibeengines.com/paper/bert.

BibTeX

@online{vibeengines-bert,
  author       = {Singh, Saurabh},
  title        = {BERT},
  year         = {2026},
  organization = {Vibe Engines},
  url          = {https://vibeengines.com/paper/bert}
}

BERT,explained.

The one-way problem

The big idea: read both ways at once

Masked language modeling: fill in the blank

Next-sentence prediction

Tokens, [CLS], and [SEP]

The recipe: pre-train once, fine-tune anywhere

110M and 340M, derived by hand

The masking recipe & the napkin math

The real results

Why it still matters

Quick answers

What is BERT?

What is masked language modeling?

How is BERT different from GPT?

What does pre-train then fine-tune mean?

Why does BERT still matter?

Go deeper on AI systems

Attention Is All You Need →

Design Semantic Search →

Design a RAG Pipeline →

Explore the topic

More Paper Breakdowns

Explore more from Vibe Engines

Get the next one in your inbox.

Cite this page

BERT,
explained.