Global
vectors.

Rung 1 learns by streaming over a corpus and sampling local windows, so it sees the co-occurrence statistics implicitly, a few pairs at a time, and many times over. This rung's change is to compute those statistics once, globally, into a co-occurrence matrix, and then fit vectors to the matrix directly.

The argument for why that is the right target is the rung's actual content. Raw co-occurrence counts are a poor thing to model — "ice" and "steam" both co-occur heavily with "water", which says little. What discriminates is the ratio: "solid" co-occurs far more with "ice" than with "steam", and "gas" the other way round, while a neutral word like "water" has a ratio near one. Meaning lives in ratios of co-occurrence probabilities, and since a ratio is a difference of logarithms, the natural objective is to make a dot product of two vectors recover the log co-occurrence count. That is what turns an intuition into a loss.

Two details matter in practice. The fit is weighted least squares rather than plain, because the matrix spans many orders of magnitude and the very frequent pairs would otherwise dominate everything — the weighting function rises and then flattens, so common pairs stop gaining influence past a threshold and the extremely rare ones, which are mostly noise, are discounted. And because the statistics are gathered first, training is a fit to a fixed object rather than a pass over text, which parallelises differently.

The distributional hypothesis says a word's meaning is revealed by the words it appears near. Make that concrete: scan a huge corpus and build a co-occurrence matrix X, where X_ij is the number of times word j shows up in the context of word i. That matrix is enormous and sparse, but it holds the entire distributional signature of the vocabulary — everything statistics can tell you about how words relate.

GloVe (Global Vectors, Pennington, Socher & Manning, 2014) turns that table into short dense vectors. It learns a vector for each word such that the dot product of two word vectors (plus a couple of bias terms) approximates the logarithm of their co-occurrence count. Fit that across all nonzero entries and you get embeddings where similar words cluster and where vector differences encode relationships. Unlike word2vec, which streams the corpus predicting local windows, GloVe aggregates the whole corpus into X first and then factorizes it — global statistics, used explicitly.

Why fit the log of the counts, rather than the counts themselves? GloVe's central observation is that the useful information isn't in a single co-occurrence probability but in the ratio of two. Take the words ice and steam. Both co-occur with water (they're both about H₂O), and both co-occur rarely with fashion. Neither raw count distinguishes them. But probe with solid: it appears much more with ice than with steam. Probe with gas: the reverse. The ratio P(solid | ice) / P(solid | steam) is large; P(gas | ice) / P(gas | steam) is small. Those ratios are where the meaning lives.

Ratios are multiplicative, and a vector space is additive — dot products add and subtract, they don't divide. Taking logs bridges the two: a ratio of probabilities becomes a difference of logs, and a difference of logs can be modeled by a difference of dot products. So fitting w_i · w̃_j ≈ log(X_ij) is precisely the design that lets vector arithmetic capture co-occurrence ratios — and, as a bonus, the linear analogies (king − man + woman ≈ queen) that made word embeddings famous. The log isn't cosmetic; it's the hinge that turns statistics into geometry.

A plain least-squares fit on log(X_ij) has two problems, and one weighting function f(x) fixes both:

Without weighting, the two extremes wreck the fit. Extremely common pairs — every content word with the, of, and — would contribute overwhelming error and pull all the vectors toward modeling function words. Meanwhile rare pairs, whose counts are just noise, would be treated as equally important as reliable ones. The weighting f(x) ramps up for rarer-but-real pairs, saturates to 1 above a cap so hyper-frequent pairs can't dominate, and — the quietly crucial part — is exactly zero at x = 0, so the loss only ever sums over the small fraction of word pairs that actually co-occur. That's what makes fitting a vocabulary-squared matrix tractable.

GloVe minimizes a single weighted least-squares loss over all word pairs:

Each term pulls the dot-product-plus-biases toward the log co-occurrence, weighted by f(X_ij). w_i and w̃_j are the "word" and "context" vectors (the final embedding is usually their sum); b_i, b̃_j are learned biases that absorb per-word frequency.

Read it as a regression: the target is log X_ij, the prediction is w_i·w̃_j + b_i + b̃_j, the residual is squared, and each residual is scaled by f(X_ij). Two learned bias terms per word soak up the fact that some words are just more frequent than others, so the dot product is free to capture the interesting relational structure. Training is stochastic gradient descent with the AdaGrad step-size rule over only the nonzero entries. The runnable version below computes the weighting, the prediction, and a single weighted loss term — the whole objective in miniature.

GloVe (Pennington, Socher and Manning, EMNLP 2014) runs in two phases that never overlap: one pass over the corpus to count, then many passes over the counts to fit. The fitting phase never looks at text again, which is the practical difference from word2vec, a method that keeps re-reading the corpus window by window.

1. Tokenise and fix the vocabulary. The paper lowercased and tokenised each corpus with the Stanford tokenizer and kept the 400,000 most frequent words. Corpora ranged from a one-billion-token 2010 Wikipedia dump to 42 billion tokens of Common Crawl (an 840-billion-token corpus was trained too, without lowercasing), and a six-billion-token combination of Wikipedia 2014 and Gigaword 5 was used for most comparisons.
2. Count with a distance-weighted window. A symmetric window of ten words on each side slides over the text, and a pair d words apart adds 1/d to X_ij rather than 1, so neighbours count fully and words at the edge of the window count a tenth. Left-only windows were also tested. Because counting is symmetric, X is symmetric.
3. Keep only the non-zeros. Most of the 400,000 × 400,000 possible pairs never co-occur, and the weighting function gives them zero weight, so they are never stored or visited. Modelling counts as a power law in the rank of the pair, the paper estimates that the number of non-zero entries grows like |C|^^0.8 for a corpus of |C| tokens: sublinear, so for a large corpus an iteration over the counts is cheaper than an epoch over the text.
4. Initialise two tables. Every word gets a word vector w_i, a context vector w̃_i and a scalar bias for each, all random. Since X is symmetric the two tables play identical roles and differ only by where they start.
5. Fit with AdaGrad. Sample non-zero entries in random order and take an AdaGrad step on each weighted squared residual, starting from a learning rate of 0.05, with x_max = 100 and α = 3/4 in the weighting function. The paper ran 50 iterations for vectors under 300 dimensions and 100 otherwise.
6. Sum the tables. The released vectors are W + W̃. The paper reports that the sum typically gives a small boost — largest on the semantic analogies — and likens it to combining several trained networks to reduce noise.
7. Evaluate like word2vec. The test bed was word2vec's 19,544-question analogy set, answered by vector offset and cosine, plus five word-similarity datasets and named-entity recognition with the vectors as features.

Why this objective rather than word2vec's

The paper's most useful derivation connects the two methods. Sum skip-gram's softmax log-likelihood over a whole corpus and group identical pairs, and the objective becomes Σ_i X_i H(P_i, Q_i): a cross-entropy between each word's empirical context distribution P_i and the model's Q_i, weighted by the word's total count X_i. The authors object to two things in it. Cross-entropy on a long-tailed distribution gives too much weight to unlikely events, and Q has to be normalised over the whole vocabulary. Replace it with a least-squares fit on unnormalised log values, swap the fixed weight X_i for a tunable f(X_ij), and the result is exactly GloVe's loss.

• Pik = Xik / Xi — the probability that word k appears in the context of word i, where X_i is the row total.
• (wi − wj) · w̃k — the ratio compares two words, so the model is asked to use their vector difference; taking a dot product with the probe's context vector keeps the result a scalar without mixing dimensions.
• F = exp — the authors require F to turn a difference of dot products into a quotient of probabilities, F(a − b) = F(a)/F(b), and the exponential is the function that does that.
• bi, b̃k — log X_i does not depend on k, so it is absorbed into a per-word bias, and the second bias restores the symmetry between words and contexts.

The last line is asked of every pair at once and cannot hold exactly, so it is fitted by weighted least squares over the non-zero X_ik, which is the objective J shown earlier on this page. At the extremes: where X_ik = 0 the target log 0 is undefined, one reason the weight must vanish there; weighting every pair equally would hand the fit to the huge number of rare, noisy pairs, and weighting by raw counts would hand it to pairs involving "the" and "of".

The authors evaluated on three kinds of task: word2vec's analogy set, five word-similarity datasets (WordSim-353, MC, RG, SCWS and Rare Words), and CoNLL-2003 named-entity recognition with the vectors as features. Baselines included SVD on the same counts, raw and with log or square-root scaling, and word2vec's CBOW and skip-gram, which the authors retrained themselves on the same six-billion-token corpus with the same vocabulary and window so that only the method differed.

The two 42-billion-token rows are the scaling result: plain SVD got worse with seven times more data while GloVe kept improving. The ablations were informative too. Accuracy rose with dimension but showed diminishing returns beyond about 200. Small, left-only windows helped syntactic questions and wider windows helped semantic ones, since meaning is often non-local. And the smaller Wikipedia corpora beat the larger Gigaword corpus on semantic questions, which the authors attribute to Wikipedia's coverage of the places those questions ask about.

Plotted against training time, GloVe beat the retrained word2vec at every budget, and in the authors' runs adding negative samples beyond about ten made word2vec worse. That comparison did not settle the matter. Levy, Goldberg and Dagan later matched hyperparameters across count-based and prediction-based methods and found mostly local or insignificant differences, with no method ahead across the board.

The cost is stated: the co-occurrence matrix must be built and held, which is memory proportional to the number of observed pairs rather than to the vocabulary, and is substantial for a large corpus. Rung 1 needs no such structure.

The comparison between the two was contested and the resolution is the useful part. Levy and Goldberg showed that rung 1's objective is implicitly factorising a shifted pointwise-mutual-information matrix — so the two methods are doing much the same thing, one explicitly and one implicitly — and Levy, Goldberg and Dagan later found that reported quality differences owed more to hyperparameters and preprocessing than to the objectives. That finding generalises well beyond word vectors, and it is the same lesson the PPO reproducibility studies teach on another ladder: two methods compared as published are often two configurations compared.

How it held up: the released vectors were the default text representation for several years, which is a long time, and then contextual embeddings made static per-word vectors obsolete for nearly everything. What survives is the framing — state what quantity your representation should recover, then fit it — and the sharpest way to see this rung's contribution is that it made the target explicit where rung 1 left it implicit.

For several years, GloVe and word2vec were the way to represent words, and pretrained GloVe vectors were a default first layer in a huge fraction of NLP models — sentiment classifiers, taggers, early question-answering systems. They proved that meaning could be distilled from raw co-occurrence statistics into a compact geometry, cheaply and reusably, and they popularized the analogy demonstrations that made embeddings intuitive to a wide audience.

The field then moved to contextual embeddings, where a word's vector depends on the sentence it's in — ELMo, then BERT and the Transformer, which resolve "bank" differently in "river bank" and "bank account." Those largely superseded static vectors for high-accuracy tasks. But GloVe's lesson endures inside them: the input embedding tables of every modern language model still map tokens to dense vectors learned from distributional statistics — the same idea, now trained end-to-end and made context-dependent. GloVe remains one of the clearest on-ramps to understanding how text becomes numbers a network can reason over.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The longest ladder here, and the most linear: nine papers that carry one idea — text as geometry — from a 2013 word embedding to a graph-structured retrieval system.

1. 1Word2Vec2013
2. 2GloVe2014
3. 3Sentence-BERT2019
4. 4Product Quantization2011
5. 5HNSW2016
6. 6Dense Passage Retrieval2020
7. 7ColBERT2020
8. 8Retrieval-Augmented Generation2020
9. 9GraphRAG2024

All 110 papers, in 19 ladders →