The Meaning Map.

How embeddings work — four ideas

1. A word is a vector. Each word maps to a list of numbers, learned so words in similar contexts get similar vectors.
2. Similarity is cosine. Compare two words by the angle between their vectors, not their distance — direction carries the meaning.
3. Neighbours are meaning. The nearest vectors to a word are its closest relatives: cat → kitten, dog; tokyo → japan.
4. Analogies are arithmetic. Concepts become consistent directions, so king − man + woman shifts along the gender axis and lands on queen.

The whole thing in code

import numpy as np

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

def nearest(word, embeds, k=3):
    v = embeds[word]
    scored = [(w, cosine(v, u)) for w, u in embeds.items()
              if w != word]
    return sorted(scored, key=lambda t: -t[1])[:k]

def analogy(a, b, c, embeds):        # a - b + c
    target = embeds[a] - embeds[b] + embeds[c]
    scored = [(w, cosine(target, u)) for w, u in embeds.items()
              if w not in (a, b, c)]
    return max(scored, key=lambda t: t[1])   # → queen

How the space is learned, then searched

1. Collect context pairs. Slide a window over the text (5 words on each side is a common default) and record every (word, neighbor) pair as a positive example.
2. Add negatives. For each positive pair, draw k random words (k = 5 is typical), sampled in proportion to frequency^0.75 so the most common words do not swamp the sample.
3. Score with a dot product and train. Word2vec's skip-gram loss pushes σ(u·v) toward 1 for real pairs and toward 0 for the negatives. Words that share contexts get the same pushes, so their vectors end up pointing the same way.
4. Search by cosine. Embed the query and rank stored vectors by cos(a, b). Once every vector has length 1, cosine is just the dot product, and ||a − b||² = 2 − 2·cos(a, b), so Euclidean distance gives the same ranking.

# skip-gram, negative sampling
loss = −log σ(u_c · v_w)
       − Σ_n log σ(−u_n · v_w)
# similarity of two vectors
cos(a, b) = a · b / (|a| · |b|)

Worked example: this page's vectors

1. The axes. Each word here is a 0/1 vector over 11 readable dimensions: royal, male, female, animal, cat, dog, young, city, country, Europe, Asia. cat is animal + cat; kitten is animal + cat + young.
2. cat vs kitten. Dot product 2, lengths √2 and √3, so cos = 2/√6 ≈ 0.82, cat's top neighbor.
3. kitten vs puppy, cat vs dog. Sharing animal + young gives 2/3 ≈ 0.67; sharing only animal gives 1/2 = 0.50.
4. king's neighbors. Click king and the nearest word is man (0.71), not queen (0.50). Each shares one axis with king, but man has no second axis to dilute the match.
5. The analogies. king − man + woman = royal + female, which is exactly queen: cosine 1.00. paris − france + japan = city + Asia, exactly tokyo: also 1.00.
6. Why the inputs are excluded. Against that same king − man + woman target, woman scores 0.71 and king 0.50. In real word2vec vectors an input word often scores highest, so analogy tests skip a, b and c, as the code above does.