Vibe Engines Learn
LABS · 07  /  ATTENTION

The Spotlight.

Inside a Transformer, every word gets to look at every other word and decide how much to listen to each one. Click a word and watch its spotlight fall — the verb leaning on its subject, the determiner pointing at its noun — as glowing links whose weights add up to one.

THE GIST · 20 SECONDS

Self-attention is how a Transformer moves meaning between words. Each token asks a query, compares it to every token's key to get scores, and a softmax turns those into weights that sum to 1. The token's new value is a weighted blend of all the values. So "sat" can pull in "cat" (its subject) and "mat" (its place) — context, computed.

  • Querywhat a word looks for
  • Keywhat a word offers
  • Weighthow much to attend
  • Sum = 1a softmax split

Click any word to make it the query — the links and shading show how its attention is split across the sentence.

UNDER THE HOOD

What you just played, written down

Every token blends in a little of every other token, weighted by relevance. That weighted blend — computed for all tokens at once — is the whole reason Transformers understand context.

How attention works — four moves

  1. Three projections. Each token becomes a query (what I want), a key (what I offer), and a value (what I carry).
  2. Score every pair. A token's query is compared to every key by dot product — high score means "very relevant to me".
  3. Softmax to weights. Scores go through a softmax, becoming non-negative weights that sum to 1 — an attention budget.
  4. Blend the values. The token's output is the weighted sum of all the values, so it absorbs context from the words it attends to.
WHAT YOU'RE SEEING

The links and bars show the softmax weights — the "attends to" part of attention. The full operation then multiplies those weights by the value vectors and sums them: softmax(QKᵀ/√d)·V.

The whole thing in code

import numpy as np

def attention(Q, K, V):
    d = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d)      # query · key, scaled
    weights = softmax(scores, axis=-1) # sum to 1 per row
    return weights @ V                 # blend the values

def softmax(x, axis=-1):
    e = np.exp(x - x.max(axis, keepdims=True))
    return e / e.sum(axis, keepdims=True)
WEIGHTSsum to 1softmax over keys
SCALE÷ √dkeeps softmax stable
MANY HEADS, MANY VIEWS

Real models run several attention heads in parallel, each with its own projections. One head may track subjects and verbs, another coreference or position — then their outputs are combined.

⚠ It's quadratic

Every token attends to every token, so cost grows with the square of the sequence length. That's why long context is expensive — and why FlashAttention and linear-attention variants exist.

↔ Self vs cross

Self-attention: queries and keys come from the same sequence (what you see here). Cross-attention: queries come from one sequence, keys/values from another — how a decoder reads an encoder.

★ Where it's used

The core of every Transformer — GPT, BERT, Claude, and Vision Transformers — plus modern speech and multimodal models. Attention is the mechanism the whole era is built on.

QUICK CHECK

Did it stick?

FAQ

Attention, answered

What is self-attention?

A mechanism where every token looks at every other token and weights how much to draw from each. A token's new representation is a weighted blend of all tokens' values, with the weights from softmax over query·key scores.

What are query, key, and value?

Three projections of each token: the query (what it's looking for), the key (what it offers), and the value (what it contributes). Scores are query·key; the output is the softmax-weighted sum of values.

Why do the weights sum to one?

A softmax turns the raw scores into a probability distribution — non-negative and summing to 1 — so each token's output is a proper weighted average.

Why divide by √d?

Dot products grow with dimension and can make the softmax too peaked (tiny gradients). Scaling by √d keeps scores in a stable range — the "scaled" in scaled dot-product attention.

What is multi-head attention?

Several attention computations in parallel, each with its own projections. Different heads capture different relationships, and their outputs are concatenated and combined.

RESULT

—

Next → stack these attention layers dozens deep, add feed-forward blocks, and train on the internet — that's a Transformer, the engine behind every modern LLM.

Finished this one? 0 / 12 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering

More Labs

Explore more from Vibe Engines

Handbooks →
Roadmaps →
System Designs →
AI System Designs →
Algorithms →
Paper Breakdowns →