BERT,
explained.

The Transformer had already shown that attention could replace recurrence. But the language models built on it still read like someone with a piece of paper covering the rest of the page: strictly left to right, one word at a time, predicting the next word from everything before it.

That one-way habit throws away half the clues. Take the word "bank" in "I went to the bank to deposit my" — you can't be sure it's a riverbank or a money bank until you read the word that comes after it. A left-to-right model, by design, never gets to peek forward. The meaning it builds is always half-blind.

BERT's bet was simple to state and awkward to pull off: let every word see the words on both sides of it, simultaneously. That's what the "B" stands for — Bidirectional Encoder Representations from Transformers.

The catch is a chicken-and-egg problem. If you train a model to predict the next word while also letting it look ahead, it can just read the answer — the future word it's supposed to guess is sitting right there. True bidirectional context and next-word prediction can't coexist. So BERT threw out next-word prediction entirely and invented a different game.

Instead of predicting the next word, BERT plays fill-in-the-blank. Take a sentence, randomly hide about 15% of its words behind a [MASK] token, and train the model to recover what was hidden — using every other word, on both sides, as evidence. This is masked language modeling (MLM).

Because the answer is now hidden rather than sitting one step ahead, the model is free to use the future as well as the past. To guess "mat," it leans on "cat," "sat," and "on" all at once. Do this across billions of words and the model is forced to learn deep, two-sided representations of meaning.

Filling in words teaches BERT about individual sentences. But many real tasks — question answering, natural-language inference — hinge on the relationship between two sentences. So BERT added a second pre-training task: next-sentence prediction (NSP). Show the model two sentences and ask a yes/no question — did B actually follow A in the original text, or is it a random impostor pulled from somewhere else?

Half the training pairs are genuine neighbors, half are random. Learning to tell them apart nudges BERT toward understanding discourse and coherence, not just word-level patterns. (Later work — RoBERTa — showed NSP was largely unnecessary, but it was part of the original recipe.)

BERT wraps its input in a few special tokens. Every sequence starts with [CLS] — a slot whose final vector becomes a summary of the whole input, perfect to hang a classifier on. Two sentences are divided by a [SEP] separator, and each token also carries a segment embedding marking whether it belongs to sentence A or B.

Words themselves are split into WordPiece sub-word units, so rare words like "tokenization" become "token" + "##ization" — a fixed 30,000-token vocabulary that never hits an unknown word. Each token's final embedding is the sum of three signals: what the word is, which segment it's in, and where it sits.

This is BERT's real gift to the field. Pre-training — the expensive part — happens once: MLM plus NSP over BooksCorpus and English Wikipedia, days of compute on a big cluster. The result is a model that already understands language in general.

Then, for any specific task, you fine-tune: bolt a tiny task-specific layer onto the [CLS] output (or every token, for tagging), and train briefly on a small labeled dataset. Sentiment, question answering, named-entity recognition — same pre-trained body, different lightweight head, a few hours each. Teams that couldn't dream of training a language model from scratch could now reach state-of-the-art by fine-tuning BERT.

BERT didn't just edge ahead — it swept. On the GLUE benchmark of nine language-understanding tasks, BERT-large pushed the score to 80.5%, a 7.7-point jump over the previous best. On the SQuAD question-answering test it passed human performance, and it set new records on eleven tasks at once.

The two sizes tell their own story: BERT-large, with 24 Transformer-encoder layers and 340M parameters, beat BERT-base everywhere — an early, clear signal that with the right pre-training, bigger keeps helping.

BERT is the encoder half of the family tree. Where GPT took the Transformer's decoder and ran toward generation, BERT took the encoder and ran toward understanding — and for years, if you were classifying, retrieving, or extracting, you reached for BERT or one of its descendants (RoBERTa, DistilBERT, ELECTRA, DeBERTa). Its sentence embeddings still power a huge share of search and retrieval systems today.

More than any single benchmark, BERT's legacy is the recipe it normalized: pre-train a Transformer on oceans of unlabeled text, then fine-tune. That idea — not the specific masking trick — is what every model after it inherited.