Why do attention weights sum to one?

The raw scores from query·key are passed through a softmax, which exponentiates and normalizes them so they form a probability distribution — non-negative and summing to one. That way each token's output is a proper weighted average of the other tokens' values, with the weights showing how its 'attention budget' is split.

Why is attention divided by the square root of the dimension?

The dot product of two d-dimensional vectors grows with d, which can push the softmax into very peaked regions with tiny gradients. Dividing the scores by the square root of the key dimension keeps them in a stable range, so training stays smooth. That is the 'scaled' in scaled dot-product attention.

LABS · 07 / ATTENTION

The Spotlight.

Q: What is self-attention?

Self-attention lets every token in a sequence look at every other token and decide how much to draw from each. Each token produces a query, and it is compared against every token's key to get a score; a softmax turns those scores into weights that sum to one, and the token's new representation is a weighted blend of all the value vectors. It is how a Transformer mixes context between words.

Q: What are query, key, and value?

Each token is projected into three vectors. The query represents what this token is looking for; the key represents what each token offers; and the value is the information a token contributes if attended to. Attention scores come from query·key, and the output is the softmax-weighted sum of the values.

Q: What is multi-head attention?

Instead of one attention computation, a Transformer runs several in parallel, each with its own query/key/value projections — these are the heads. Different heads learn to focus on different relationships (syntax, coreference, position), and their outputs are concatenated and combined, giving the model several views of context at once.

Inside a Transformer, every word gets to look at every other word and decide how much to listen to each one. Click a word and watch its spotlight fall — the verb leaning on its subject, the determiner pointing at its noun — as glowing links whose weights add up to one.

THE GIST · 20 SECONDS

Self-attention is how a Transformer moves meaning between words. Each token asks a query, compares it to every token's key to get scores, and a softmax turns those into weights that sum to 1. The token's new value is a weighted blend of all the values. So "sat" can pull in "cat" (its subject) and "mat" (its place) — context, computed.

Querywhat a word looks for
Keywhat a word offers
Weighthow much to attend
Sum = 1a softmax split

Click any word to make it the query — the links and shading show how its attention is split across the sentence.

UNDER THE HOOD

What you just played, written down

Every token blends in a little of every other token, weighted by relevance. That weighted blend — computed for all tokens at once — is the whole reason Transformers understand context.

How attention works — four moves

Three projections. Each token becomes a query (what I want), a key (what I offer), and a value (what I carry).
Score every pair. A token's query is compared to every key by dot product — high score means "very relevant to me".
Softmax to weights. Scores go through a softmax, becoming non-negative weights that sum to 1 — an attention budget.
Blend the values. The token's output is the weighted sum of all the values, so it absorbs context from the words it attends to.

WHAT YOU'RE SEEING

The links and bars show the softmax weights — the "attends to" part of attention. The full operation then multiplies those weights by the value vectors and sums them: softmax(QKᵀ/√d)·V. These particular numbers are a fixed, hand-picked example built to look like real trained attention, not a live Q·K computation over real query/key vectors — the shape (weight sums to 1, a word leans on the words it depends on) is what a real model actually does.

The whole thing in code

import numpy as np

def attention(Q, K, V):
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)      # query · key, scaled
    weights = softmax(scores, axis=-1) # sum to 1 per row
    return weights @ V                 # blend the values

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis, keepdims=True))
    return e / e.sum(axis, keepdims=True)

WEIGHTSsum to 1softmax over keys

SCALE÷ √dkeeps softmax stable

MANY HEADS, MANY VIEWS

Real models run several attention heads in parallel, each with its own projections. One head may track subjects and verbs, another coreference or position — then their outputs are combined.

⚠ It's quadratic

Every token attends to every token, so cost grows with the square of the sequence length. That's why long context is expensive — and why FlashAttention and linear-attention variants exist.

↔ Self vs cross

Self-attention: queries and keys come from the same sequence (what you see here). Cross-attention: queries come from one sequence, keys/values from another — how a decoder reads an encoder.

★ Where it's used

The core of every Transformer — GPT, BERT, Claude, and Vision Transformers — plus modern speech and multimodal models. Attention is the mechanism the whole era is built on.

QUICK CHECK

Did it stick?

FAQ

Attention, answered

What is self-attention?

A mechanism where every token looks at every other token and weights how much to draw from each. A token's new representation is a weighted blend of all tokens' values, with the weights from softmax over query·key scores.

What are query, key, and value?

Three projections of each token: the query (what it's looking for), the key (what it offers), and the value (what it contributes). Scores are query·key; the output is the softmax-weighted sum of values.

Why do the weights sum to one?

A softmax turns the raw scores into a probability distribution — non-negative and summing to 1 — so each token's output is a proper weighted average.

Why divide by √d?

Dot products grow with dimension and can make the softmax too peaked (tiny gradients). Scaling by √d keeps scores in a stable range — the "scaled" in scaled dot-product attention.

What is multi-head attention?

Several attention computations in parallel, each with its own projections. Different heads capture different relationships, and their outputs are concatenated and combined.

Finished this one? 0 / 61 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering