Global
vectors.

The distributional hypothesis says a word's meaning is revealed by the words it appears near. Make that concrete: scan a huge corpus and build a co-occurrence matrix X, where X_ij is the number of times word j shows up in the context of word i. That matrix is enormous and sparse, but it holds the entire distributional signature of the vocabulary — everything statistics can tell you about how words relate.

GloVe (Global Vectors, Pennington, Socher & Manning, 2014) turns that table into short dense vectors. It learns a vector for each word such that the dot product of two word vectors (plus a couple of bias terms) approximates the logarithm of their co-occurrence count. Fit that across all nonzero entries and you get embeddings where similar words cluster and where vector differences encode relationships. Unlike word2vec, which streams the corpus predicting local windows, GloVe aggregates the whole corpus into X first and then factorizes it — global statistics, used explicitly.

Why fit the log of the counts, rather than the counts themselves? GloVe's central observation is that the useful information isn't in a single co-occurrence probability but in the ratio of two. Take the words ice and steam. Both co-occur with water (they're both about H₂O), and both co-occur rarely with fashion. Neither raw count distinguishes them. But probe with solid: it appears much more with ice than with steam. Probe with gas: the reverse. The ratio P(solid | ice) / P(solid | steam) is large; P(gas | ice) / P(gas | steam) is small. Those ratios are where the meaning lives.

Ratios are multiplicative, and a vector space is additive — dot products add and subtract, they don't divide. Taking logs bridges the two: a ratio of probabilities becomes a difference of logs, and a difference of logs can be modeled by a difference of dot products. So fitting w_i · w̃_j ≈ log(X_ij) is precisely the design that lets vector arithmetic capture co-occurrence ratios — and, as a bonus, the linear analogies (king − man + woman ≈ queen) that made word embeddings famous. The log isn't cosmetic; it's the hinge that turns statistics into geometry.

A plain least-squares fit on log(X_ij) has two problems, and one weighting function f(x) fixes both:

Without weighting, the two extremes wreck the fit. Extremely common pairs — every content word with the, of, and — would contribute overwhelming error and pull all the vectors toward modeling function words. Meanwhile rare pairs, whose counts are just noise, would be treated as equally important as reliable ones. The weighting f(x) ramps up for rarer-but-real pairs, saturates to 1 above a cap so hyper-frequent pairs can't dominate, and — the quietly crucial part — is exactly zero at x = 0, so the loss only ever sums over the small fraction of word pairs that actually co-occur. That's what makes fitting a vocabulary-squared matrix tractable.

GloVe minimizes a single weighted least-squares loss over all word pairs:

Each term pulls the dot-product-plus-biases toward the log co-occurrence, weighted by f(X_ij). w_i and w̃_j are the "word" and "context" vectors (the final embedding is usually their sum); b_i, b̃_j are learned biases that absorb per-word frequency.

Read it as a regression: the target is log X_ij, the prediction is w_i·w̃_j + b_i + b̃_j, the residual is squared, and each residual is scaled by f(X_ij). Two learned bias terms per word soak up the fact that some words are just more frequent than others, so the dot product is free to capture the interesting relational structure. Training is plain gradient descent (AdaGrad in the paper) over only the nonzero entries. The runnable version below computes the weighting, the prediction, and a single weighted loss term — the whole objective in miniature.

GloVe unified count-based and prediction-based embeddings and produced strong, reusable vectors:

The limitations are the limitations of all static embeddings: one vector per word type, so "bank" (river) and "bank" (money) collapse into a single point, and no sensitivity to sentence context. Building the global matrix also costs memory. But within the static-embedding world, GloVe was a clean, well-motivated, top-performing method — and a clear demonstration that classic matrix factorization on co-occurrence counts and neural prediction were two views of the same thing.

For several years, GloVe and word2vec were the way to represent words, and pretrained GloVe vectors were a default first layer in a huge fraction of NLP models — sentiment classifiers, taggers, early question-answering systems. They proved that meaning could be distilled from raw co-occurrence statistics into a compact geometry, cheaply and reusably, and they popularized the analogy demonstrations that made embeddings intuitive to a wide audience.

The field then moved to contextual embeddings, where a word's vector depends on the sentence it's in — ELMo, then BERT and the Transformer, which resolve "bank" differently in "river bank" and "bank account." Those largely superseded static vectors for high-accuracy tasks. But GloVe's lesson endures inside them: the input embedding tables of every modern language model still map tokens to dense vectors learned from distributional statistics — the same idea, now trained end-to-end and made context-dependent. GloVe remains one of the clearest on-ramps to understanding how text becomes numbers a network can reason over.