Word2Vec,
explained.

Rung 1, so the comparison is against how text was represented before. A word was a slot — one dimension in a vocabulary-sized vector, on or off. Under that representation every pair of distinct words is equally dissimilar: the distance from "cat" to "dog" is the distance from "cat" to "bureaucracy". Any notion of similarity had to come from a hand-built resource like a thesaurus.

The change is to learn a dense vector per word from context alone, on the hypothesis that words appearing in similar company mean similar things. Train a tiny model to predict a word's neighbours from the word, or the word from its neighbours, and throw the model away — the weights are the product. Similarity becomes geometry, and it is computed rather than curated.

Two things to carry up the ladder. First the scale trick that made it practical: predicting over a full vocabulary with a softmax is prohibitive, so the objective is reframed as telling real context words apart from a handful of randomly-drawn fake ones — negative sampling, which turns one expensive normalisation into a few cheap binary decisions and reappears at rungs 3 and 6. Second the result that made it famous: the space has linear structure, so vector arithmetic on king − man + woman lands near queen. Nothing in the objective asked for that, and it is the first evidence on this ladder that a learned representation can carry relations it was never taught.

Early NLP treated each word as a one-hot vector — a giant list of zeros with a single 1 marking that word's slot. In that scheme every word is exactly as far from every other; "Paris" and "France" are as unrelated as "Paris" and "banana." All the meaning is thrown away.

What you really want is a representation where similar words sit near each other — where the geometry itself carries meaning. That's a dense embedding: a few hundred real numbers per word instead of a lonely 1 in a sea of zeros.

Word2Vec's foundation is an old linguistic idea, the distributional hypothesis: words that show up in similar contexts tend to mean similar things. "Coffee" and "tea" appear around "drink," "cup," "hot," "morning" — so if you learn to predict context, words with similar contexts are pushed toward similar vectors.

The genius was making this cheap. Word2Vec is a tiny, shallow network — no deep stacks — trained on billions of words fast enough to actually run at scale.

The paper offers two mirror-image training games. CBOW (Continuous Bag of Words) hides the middle word and predicts it from the words around it. Skip-gram flips it: given the middle word, predict the words likely to surround it.

Nobody actually cares about these predictions. They're a pretext. The real prize is a by-product: to predict context well, the network is forced to arrange word vectors so that similar words cluster — and those vectors are what you keep. CBOW is faster; skip-gram shines on rare words and huge corpora.

The first paper got to billions of words with a hierarchical softmax; the follow-up's negative sampling is the cheaper trick most people remember. Rather than a full softmax over the whole vocabulary every step, it turns training into a cheap yes/no game: push a word's vector toward its real neighbors and away from a handful of k random impostors:

Real (word, context) pairs get their dot product raised; k sampled non-neighbors get theirs lowered. That's the whole training signal.

Do that across a corpus and a geometry emerges where direction encodes meaning — which is what lets you do arithmetic on words. The runnable version below shows that arithmetic resolving analogies by cosine nearest-neighbor.

The result that made the field gasp: the vectors captured relationships as consistent directions. The step from "man" to "woman" is roughly the same vector as "king" to "queen" — a gender direction. So you can literally compute:

"Paris − France + Italy" lands near "Rome." These analogies weren't programmed in — they emerged from nothing but learning to predict nearby words. Meaning had genuine mathematical structure.

Every embedding model owes Word2Vec its core premise: meaning can be a vector, and similarity can be distance. Modern systems use contextual embeddings from Transformers — where a word's vector shifts with its sentence, so "bank" by a river differs from "bank" with money — rather than one fixed vector per word. But the leap Word2Vec made is the one everything stands on.

When you build semantic search, a recommender, or a RAG retriever, you're placing things on a map of meaning and measuring nearness. That map was first drawn here.

Word2vec is two papers. The first (Mikolov, Chen, Corrado and Dean, January 2013) proposed the CBOW and skip-gram architectures, made them cheap with a hierarchical softmax, and introduced a 19,544-question analogy test. The second (Mikolov, Sutskever, Chen, Corrado and Dean, NIPS 2013) replaced the hierarchical softmax with negative sampling, added subsampling of frequent words, and extended the model to phrases. The procedure below is the combined recipe that the released word2vec tool implements.

1. Build the vocabulary. Count every token and drop rare ones; the second paper discarded words seen fewer than five times, which left 692,000 words from a one-billion-word news corpus. Each surviving word gets two vectors: an input vector v_w, which is the one you keep, and an output vector v′_w, used only during training.
2. Slide a window. At each position the centre word is paired with words up to c positions either side. Skip-gram does not use a fixed width: for every centre word it draws R uniformly from 1 to c and takes R words on each side, so near neighbours are sampled more often than distant ones. The first paper used c = 10 for skip-gram and four words either side for CBOW.
3. Pick the prediction. CBOW averages the input vectors of the context words and predicts the centre word. Skip-gram uses the centre word's vector to predict each context word separately, so one position yields 2R training pairs. Both models are log-linear: no hidden non-linearity, just a dot product between an input vector and an output vector.
4. Avoid the full softmax. Normalising over the whole vocabulary on every pair is the expensive part, and the two papers solve it differently. The first uses a hierarchical softmax: the vocabulary becomes a binary Huffman tree built from word counts, each inner node has an output vector, and a word's probability is the product of left-or-right sigmoid decisions along its path. A balanced tree over a million words needs about 20 decisions; the Huffman tree roughly halves that on average because frequent words sit near the root. The second paper uses negative sampling: score the true context word against k noise words drawn from the unigram distribution raised to the 3/4 power, with k from 5 to 20 on small corpora and 2 to 5 on large ones.
5. Subsample frequent words. Each occurrence of word w is discarded with probability 1 − √(t / f(w)), where f(w) is its share of the corpus and t is around 10^⁻⁵. Words rarer than t are always kept, while a word as common as the loses almost every occurrence. In the released code the discarded words are removed before windowing, so the effective window widens as well.
6. Train with stochastic gradient descent. The first paper's smaller comparisons used three epochs of stochastic gradient descent with a learning rate that starts at 0.025 and decays linearly to zero. Its largest runs, on a Google News corpus of about six billion tokens with the vocabulary capped at the million most frequent words, were spread across a cluster with the DistBelief framework, asynchronous mini-batch updates and Adagrad: the 1,000-dimensional skip-gram model took roughly two and a half days on 125 CPU cores.
7. Keep the input vectors and query them. The output vectors are thrown away. Similarity is cosine between input vectors. An analogy a : b :: c : ? is answered by computing v_b − v_a + v_c and returning the nearest word by cosine, excluding the three question words, and it counts only if that word is exactly the expected one — synonyms are scored as errors.

Phrases

The second paper also learns vectors for phrases such as "New York Times". A bigram is merged into one token when (count(w_iw_j) − δ) / (count(w_i) × count(w_j)) exceeds a threshold, where the discount δ stops pairs of rare words from qualifying. Two to four passes with a falling threshold build longer phrases, which are then trained exactly like words.

• vI, v′O — the input vector of the centre word and the output vector of one context word; only the input vectors survive training.
• W — the vocabulary size. The softmax denominator touches every word, which is what makes the exact objective too slow to train on billions of pairs.
• σ(x) = 1/(1 + e^−x) — the logistic function: the true pair is a yes decision, each noise word a no.
• k — noise words per positive pair: 5 to 20 on small corpora, 2 to 5 on large ones.
• U(w)^3/4 — unigram counts raised to the 3/4 power and renormalised, which the second paper found clearly better than either the raw unigram or a uniform distribution.

Negative sampling is noise-contrastive estimation with the noise-probability correction dropped, so it no longer approximates the softmax's probabilities; it only has to produce good vectors. At the extremes: with k = 0 there is nothing to push against, and the objective is maximised by making every dot product huge; with the exponent at 1 the negatives are mostly "the" and "of", and at 0 they are mostly rare words nothing is confused with. Hierarchical softmax instead writes p(w | w_I) as a product of σ(±v′_n · v_I) over the inner nodes n on w's tree path — a properly normalised distribution at one sigmoid per level.

The first paper's test was its own analogy set: 8,869 semantic questions (capitals, currencies, cities in states, male and female forms) and 10,675 syntactic ones (plurals, tenses, comparatives, nationality adjectives), scored by exact match. It compared its two architectures with earlier neural language models at 640 dimensions on the same training data, then scaled them to the six-billion-word Google News corpus on a cluster. The second paper trained 300-dimensional skip-gram models on a one-billion-word news corpus and compared ways of avoiding the full softmax.

The architecture rows explain the usual advice: skip-gram was far stronger on semantic questions, CBOW slightly stronger on syntactic ones and cheaper to train. The second paper's ablations carry its argument. Subsampling frequent words at t = 10^⁻⁵ cut training time by roughly half or more — negative sampling with k = 5 went from 38 to 14 minutes — while holding or improving accuracy, and it lifted hierarchical softmax from 47% to 55%. The 3/4-power noise distribution beat both the raw unigram and the uniform distribution on every task.

The negative result is that no estimator won everywhere. Hierarchical softmax lost to negative sampling on single-word analogies, yet on the paper's new phrase analogies it became the best method once frequent words were subsampled. The best phrase model — hierarchical softmax, 1,000 dimensions, the whole sentence as context and about 33 billion training words — reached 72%, against 66% with six billion words.

The structural limitation is one vector per word, forever: a word has a single representation regardless of context, so every sense of "bank" is averaged into one point. The model has no mechanism to do otherwise, and the paper does not claim one.

The analogy result was also over-read, and the correction is instructive. Later analyses showed the arithmetic works much less cleanly than the famous examples suggest — performance depends heavily on the evaluation excluding the query words themselves, and many relation types barely work at all. The linear structure is real and it is weaker than the demonstration implied. Worth holding as a general caution: a striking example is not a measurement.

The other finding that aged badly in a more serious way: because the vectors encode the statistics of the corpus, they encode its biases as geometry, which made "similarity is just geometry" a claim with consequences nobody had priced in.

How it held up: the specific model is history and the idea is the foundation of everything above it. Every rung on this ladder, and every retrieval system on this site, rests on "meaning is a direction in a learned space". The one-vector-per-word limit was removed by contextual models — which is rung 1 of the scaling ladder, and why rung 3 here is about repairing it for similarity search.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The longest ladder here, and the most linear: nine papers that carry one idea — text as geometry — from a 2013 word embedding to a graph-structured retrieval system.

1. 1Word2Vec2013
2. 2GloVe2014
3. 3Sentence-BERT2019
4. 4Product Quantization2011
5. 5HNSW2016
6. 6Dense Passage Retrieval2020
7. 7ColBERT2020
8. 8Retrieval-Augmented Generation2020
9. 9GraphRAG2024

All 110 papers, in 19 ladders →