Word2Vec,
explained.

Rung 1, so the comparison is against how text was represented before. A word was a slot — one dimension in a vocabulary-sized vector, on or off. Under that representation every pair of distinct words is equally dissimilar: the distance from "cat" to "dog" is the distance from "cat" to "bureaucracy". Any notion of similarity had to come from a hand-built resource like a thesaurus.

The change is to learn a dense vector per word from context alone, on the hypothesis that words appearing in similar company mean similar things. Train a tiny model to predict a word's neighbours from the word, or the word from its neighbours, and throw the model away — the weights are the product. Similarity becomes geometry, and it is computed rather than curated.

Two things to carry up the ladder. First the scale trick that made it practical: predicting over a full vocabulary with a softmax is prohibitive, so the objective is reframed as telling real context words apart from a handful of randomly-drawn fake ones — negative sampling, which turns one expensive normalisation into a few cheap binary decisions and reappears at rungs 3 and 6. Second the result that made it famous: the space has linear structure, so vector arithmetic on king − man + woman lands near queen. Nothing in the objective asked for that, and it is the first evidence on this ladder that a learned representation can carry relations it was never taught.

Early NLP treated each word as a one-hot vector — a giant list of zeros with a single 1 marking that word's slot. In that scheme every word is exactly as far from every other; "Paris" and "France" are as unrelated as "Paris" and "banana." All the meaning is thrown away.

What you really want is a representation where similar words sit near each other — where the geometry itself carries meaning. That's a dense embedding: a few hundred real numbers per word instead of a lonely 1 in a sea of zeros.

Word2Vec's foundation is an old linguistic idea, the distributional hypothesis: words that show up in similar contexts tend to mean similar things. "Coffee" and "tea" appear around "drink," "cup," "hot," "morning" — so if you learn to predict context, words with similar contexts are pushed toward similar vectors.

The genius was making this cheap. Word2Vec is a tiny, shallow network — no deep stacks — trained on billions of words fast enough to actually run at scale.

The paper offers two mirror-image training games. CBOW (Continuous Bag of Words) hides the middle word and predicts it from the words around it. Skip-gram flips it: given the middle word, predict the words likely to surround it.

Nobody actually cares about these predictions. They're a pretext. The real prize is a by-product: to predict context well, the network is forced to arrange word vectors so that similar words cluster — and those vectors are what you keep. CBOW is faster; skip-gram shines on rare words and huge corpora.

The engine that made it fast enough for billions of words is negative sampling. Rather than a full softmax over the whole vocabulary every step, it turns training into a cheap yes/no game: push a word's vector toward its real neighbors and away from a handful of k random impostors:

Real (word, context) pairs get their dot product raised; k sampled non-neighbors get theirs lowered. That's the whole training signal.

Do that across a corpus and a geometry emerges where direction encodes meaning — which is what lets you do arithmetic on words. The runnable version below shows that arithmetic resolving analogies by cosine nearest-neighbor.

The result that made the field gasp: the vectors captured relationships as consistent directions. The step from "man" to "woman" is roughly the same vector as "king" to "queen" — a gender direction. So you can literally compute:

"Paris − France + Italy" lands near "Rome." These analogies weren't programmed in — they emerged from nothing but learning to predict nearby words. Meaning had genuine mathematical structure.

Word2Vec was fast, simple, and stunningly good. It set records on word-analogy and similarity benchmarks, trained on billions of words on modest hardware, and produced vectors you could just download and plug into any NLP system. It turned "represent words as vectors" from a research curiosity into standard practice almost overnight.

Every embedding model owes Word2Vec its core premise: meaning can be a vector, and similarity can be distance. Modern systems use contextual embeddings from Transformers — where a word's vector shifts with its sentence, so "bank" by a river differs from "bank" with money — rather than one fixed vector per word. But the leap Word2Vec made is the one everything stands on.

When you build semantic search, a recommender, or a RAG retriever, you're placing things on a map of meaning and measuring nearness. That map was first drawn here.

The structural limitation is one vector per word, forever: a word has a single representation regardless of context, so every sense of "bank" is averaged into one point. The model has no mechanism to do otherwise, and the paper does not claim one.

The analogy result was also over-read, and the correction is instructive. Later analyses showed the arithmetic works much less cleanly than the famous examples suggest — performance depends heavily on the evaluation excluding the query words themselves, and many relation types barely work at all. The linear structure is real and it is weaker than the demonstration implied. Worth holding as a general caution: a striking example is not a measurement.

The other finding that aged badly in a more serious way: because the vectors encode the statistics of the corpus, they encode its biases as geometry, which made "similarity is just geometry" a claim with consequences nobody had priced in.

How it held up: the specific model is history and the idea is the foundation of everything above it. Every rung on this ladder, and every retrieval system on this site, rests on "meaning is a direction in a learned space". The one-vector-per-word limit was removed by contextual models — which is rung 1 of the scaling ladder, and why rung 3 here is about repairing it for similarity search.

Word2Vec's specific architecture — a shallow network trained with skip-gram or CBOW and negative sampling — has largely been superseded. But the idea it proved is now the foundation of modern AI: that meaning can be represented as a dense vector, learned from context, where geometric closeness encodes semantic similarity. Every embedding model, every vector database, and the input layer of every Transformer descends from this insight.

The famous king − man + woman ≈ queen result was the moment the field saw that these vectors captured relationships, not just similarity — that analogies lived as consistent directions in the space. It was a striking, almost surprising demonstration that structure emerges from nothing but the company a word keeps.

The key limitation, and the reason it was eventually replaced, is that Word2Vec gives each word one static vector regardless of context. "Bank" by a river and "bank" holding your money collapse into the same point. Contextual embeddings — ELMo, then BERT and the Transformer family — fixed exactly this by producing a different vector for a word depending on the sentence around it. That is the direct line from Word2Vec to today's models: same core idea (meaning as geometry), made context-aware.

The practical lesson outlasts the model: representation is everything. Before you can reason over data — words, images, users, products — you need to embed it in a space where distance means something. Word2Vec was the proof of concept for that entire paradigm, and "embed it first" is now the default first move in applied machine learning.

Answer each one out loud before you open it. Recalling an answer is what makes it stick; recognising one you have just read is not. Your score is kept on this device.

The longest ladder here, and the most linear: nine papers that carry one idea — text as geometry — from a 2013 word embedding to a graph-structured retrieval system.

1. 1Word2Vec2013
2. 2GloVe2014
3. 3Sentence-BERT2019
4. 4Product Quantization2011
5. 5HNSW2016
6. 6Dense Passage Retrieval2020
7. 7ColBERT2020
8. 8Retrieval-Augmented Generation2020
9. 9GraphRAG2024

All 110 papers, in 19 ladders →