Vibe Engines
YouTube

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

AI · Transformers

Position, Written as an Angle.

Self-attention has a curious blind spot: it treats a sequence as a bag of tokens. The dot product between a query and a key doesn’t care whether two words are adjacent or a hundred apart, so a raw transformer literally cannot tell “dog bites man” from “man bites dog.” Something has to inject position. Rotary Position Embedding (RoPE) — used in LLaMA, GPT-NeoX, and most modern open models — does it with an idea that is as elegant as it is practical: rotate each token’s query and key vector by an angle proportional to its position. A token at position 3 gets rotated three times as far as a token at position 1. The magic falls out of basic geometry: the dot product of two vectors depends only on the angle between them, and after rotating a query by m·θ and a key by n·θ, the angle between them shifts by exactly (n − m)·θ. So the attention score ends up depending only on the relative distance n − m between the two tokens — never on their absolute positions. Slide two tokens along the sequence and watch their attention score stay fixed as long as the gap between them does.

attention is order-blind · rotate query/key by position·θ · dot product depends on relative offset (n−m)
positional encoding

Extra information injected into a transformer so it knows the order of tokens — attention alone is order-blind.

RoPE

Rotary Position Embedding: rotate each token’s query and key vector by an angle proportional to its position.

relative position

What actually matters for many relationships — how far apart two tokens are, not their absolute indices.

rotation invariance

A dot product depends only on the angle between two vectors, so rotating both by the same amount leaves it unchanged.

rope.js — rotate by position, score by distance
Ready
Two tokens — a query at position m and a key at position n — each rotated by its position. Move them along the sequence and watch the attention score (their dot product). Try shift both: the score won’t change, because only the gap n−m matters.
0
query pos m
3
key pos n
-0.34
score (offset 3)

How it works

The reason RoPE feels almost too clean is that it turns “where is this token?” into a rotation, and rotations interact with dot products in exactly the way attention needs. Take a query vector and a key vector living in a 2-D plane. RoPE rotates the query by an angle proportional to the query’s position — m·θ — and the key by an angle proportional to the key’s position — n·θ. Now recall the one fact that makes it all work: the dot product of two vectors equals |a||b|·cos(angle between them), so it is completely determined by the angle between them, not their orientation in the plane. Rotating the query by m·θ and the key by n·θ changes the angle between them by (n − m)·θ — the difference of the positions. Therefore the attention score is a function of n − m alone: it is translation-invariant. That single property is why RoPE is so effective. It means the model learns relationships in terms of relative distance (“the adjective two words before the noun”), which generalizes far better than pinning meaning to absolute slots; it needs no learned position table, so nothing has to be trained per-position; and there is no table that runs out at a fixed length. That is not the same as working at any length: gaps longer than any seen in training are still unfamiliar, so quality drops past the trained length unless the angles are rescaled (position interpolation, YaRN). Real models don’t use a single 2-D plane — they split the head dimension into many pairs and give each pair its own rotation frequency θ (high frequencies capture fine, nearby distinctions; low frequencies capture coarse, long-range ones), which is directly analogous to how sinusoidal encodings use many wavelengths. But every one of those planes obeys the same rule you can see here: rotate by position, and the score comes out depending only on the distance.

1

Attention is order-blind

A plain dot product between a query and a key ignores where the tokens sit in the sequence. Without extra information, the model can’t distinguish word orderings — position must be injected.

2

Rotate each vector by its position

RoPE rotates the query by an angle m·θ (m = its position) and the key by n·θ (n = its position). A token twice as far along is rotated twice as much.

3

The dot product sees only the gap

Because a dot product depends only on the angle between two vectors, rotating the query by m·θ and the key by n·θ changes that angle by (n−m)·θ. The score becomes a function of the relative distance n−m alone.

✓

Relative and table-free

The score is translation-invariant: shift both tokens and it doesn’t change. So the model learns relative relationships and needs no learned position table. Going past the trained length still takes rescaled angles. Real models do this across many rotation frequencies at once.

Problem
attention ignores order
RoPE
rotate q,k by position·θ
Score depends on
relative offset n−m
Bonus
no learned position table

The code

# RoPE in one 2-D plane (real models do many planes/frequencies) def rotate(v, angle): c, s = cos(angle), sin(angle) return [v[0]*c - v[1]*s, v[0]*s + v[1]*c] q_m = rotate(q, m * theta) # query rotated by its position m k_n = rotate(k, n * theta) # key rotated by its position n score = dot(q_m, k_n) # = dot(q, rotate(k, (n - m)*theta)) # depends ONLY on the relative offset (n - m) — translation-invariant

The picture: same gap, same score

gap 110° → −0.34 gap 110° → −0.34 qk qk m=0, n=3q 0°, k 110° m=2, n=5q 60°, k 170° score vs offset n−m = 0 → 0.94 n−m = 1 → 0.64 n−m = 2 → 0.17 n−m = 3 → −0.34 n−m = 4 → −0.77 n−m = 5 → −0.98 offset 12 = offset 0
Shifting both tokens by two positions turns both arrows by 60°, so the angle between them stays 110° and the dot product stays −0.34. The table is the whole score function for this plane: one number per offset, whatever the absolute positions. With a single 30° frequency it repeats every 12 positions, which is why real models use many frequencies.

Worked example: the lab’s numbers, row by row

The lab rotates by θ = 30° per position. The query starts at 0° and the key at 20°, both of length 1, so the score is just the cosine of the angle between them. Query angle = 30° × m; key angle = 20° + 30° × n.

mnQuery angleKey angleGapScore
000°20°20°0.94
010°50°50°0.64
030°110°110°−0.34
2560°170°110°−0.34
69180°290°110°−0.34
012 (past the lab’s 9)0°380°380° = 20°0.94

Check one row by coordinates. At m = 2 the query is (cos 60°, sin 60°) = (0.500, 0.866); at n = 5 the key is (cos 170°, sin 170°) = (−0.985, 0.174). The dot product is 0.500 × −0.985 + 0.866 × 0.174 = −0.342, the same as the cosine of 110°. The last row is the catch. One plane with one frequency cannot tell offset 12 from offset 0. Real models give each pair of dimensions its own frequency, θ_i = 10000^(−2i/d) in the original RoPE. For a head of 128 dimensions the fastest pair turns 1 radian per position and repeats every 6.28 positions; the slowest repeats only every 54,410 positions. Read together, the pairs give every offset in a normal context its own pattern.

Cost and trade-offs: position schemes compared

Here Time is the extra compute and Space is the extra learned parameters. All of them are small next to attention itself; the real differences are in what the model can generalise to.

CaseTimeSpaceWhy
Learned absolute tableOne lookup and add per tokenmax_len × d parametersSimple, but has no entry for any position past max_len.
Sinusoidal absoluteOne add per token, at the inputNoneA fixed formula mixed into the token embedding once. Relative distance has to be learned.
RoPERotate q and k in every attention layer, O(d) per tokenNoneRelative position falls out of the dot product. Values are left alone.
ALiBiAdd −slope × distance to every attention scoreNoneNo rotation: a straight penalty that grows with distance. BLOOM used it.
Learned relative bias (T5)Add a learned bias per distance bucket to every scoreA small table of bucket biasesFar distances share a bucket, so the model sees “far” rather than an exact gap.
RoPE past the trained lengthSameNoneGaps the model never saw in training give unfamiliar rotation patterns, and quality usually drops sharply. Position interpolation and YaRN rescale the angles to fit the trained range, usually with a short fine-tune.

That last row is the limit on longer sequences. A gap of 5 does look the same everywhere, so RoPE generalises across where a pattern sits. It does not, by itself, generalise to gaps longer than any it was trained on. That takes the angle rescaling above.

Where this shows up in production and interviews

  • Stretching a model’s context

    Long-context versions of RoPE models are usually made by changing the angles, not the architecture: raise the base, or rescale with position interpolation or YaRN, then fine-tune briefly on long text. In Hugging Face model configs this appears as rope_theta and rope_scaling.

  • The KV cache

    Keys are rotated before they are cached, so each cached key already carries its position. Generating one more token rotates only the new query and key. Nothing in the cache is touched again.

  • Images and video

    Vision and multimodal models extend the idea to 2-D or 3-D by giving some dimension pairs the row position and others the column (or time) position. The same dot-product rule then gives relative position in each direction.

  • Interview angle

    Common questions: how can relative position come from absolute rotations? (Only the difference of angles reaches the dot product.) Why rotate q and k but not v? (Position should change who attends to whom, not the content passed along.) What breaks past the training length, and how is it fixed?

Common mistakes with RoPE

  • Rotating the values too

    Only q and k are rotated. Position should change who attends to whom; rotating v would scramble the content that gets passed along instead.

  • Pairing the wrong dimensions

    RoPE rotates dimensions in pairs, and implementations differ: some pair neighbours (0 with 1), others pair the first half with the second (0 with d/2). Loading weights trained one way into code written the other way gives a model that runs and quietly produces worse attention.

  • Rotating cached keys twice

    Keys are rotated once, at their own position, before they go into the KV cache. Rotating the whole cache again at every step shifts every cached key by the new position and breaks the relative distances.

  • Assuming it extrapolates for free

    A gap of 5 looks the same at any position, but a gap longer than anything seen in training is still new to the model. Without rescaled angles (position interpolation, YaRN) quality usually drops past the trained length.

  • Using one frequency

    With a single rotation speed, as in this lab, offset 12 lands on the same angle as offset 0 (30° × 12 = 360°). Real models spread pairs across many frequencies so that every offset in a normal context gets its own pattern.

Quick check

1. Why does a transformer need positional encoding at all?

2. How does RoPE encode a token’s position?

3. Why does the RoPE attention score depend only on the relative distance between tokens?

FAQ

What is RoPE (rotary position embedding)?

A way to give a transformer positional awareness by rotating each token’s query and key vectors by an angle proportional to the token’s position. Since the attention dot product depends only on the angle between vectors, rotating the query and key by their positions makes the score depend only on the relative distance between the two tokens. It’s used in many modern LLMs (LLaMA, GPT-NeoX, Mistral) because it encodes relative position naturally, needs no learned position table, and can be stretched to longer contexts by rescaling its angles (position interpolation, YaRN). Without that rescaling, quality still drops past the length it was trained on.

How is RoPE different from absolute positional encodings?

Absolute encodings add a position-dependent vector to each token, tagging its absolute index, and the model learns what the tags mean. RoPE instead bakes position into the query/key interaction by rotation, so the score sees the relative offset between tokens, not absolute positions. This generalizes better (many relationships are inherently relative) and needs no per-position learned parameters, so there is no table that runs out at a fixed length. Past the trained length the model still degrades unless the angles are rescaled (position interpolation, YaRN).

Why does encoding relative position help language models?

Because most linguistic structure is about how far apart things are, not their absolute index. Subject–verb agreement, an adjective modifying a nearby noun, a pronoun referring to a recent entity — these depend on relative distance and are the same whether the phrase is early or late in the text. A model encoding relative position learns such a pattern once and applies it everywhere. That is not the same as handling any length: gaps longer than any seen in training are still unfamiliar, so quality drops past the trained length unless the angles are rescaled (position interpolation, YaRN).

How does RoPE work across many dimensions, not just a 2-D plane?

A real head has a query/key dimension of 64 or 128, and RoPE splits it into coordinate pairs, treating each as a 2-D plane with its own rotation frequency θ. Some pairs rotate quickly with position (high frequency), capturing fine short-range distinctions; others rotate slowly (low frequency), capturing long-range relationships — analogous to the spectrum of wavelengths in sinusoidal encodings. Every pair still obeys the same rule (its score contribution depends only on the relative offset), so the head encodes relative position across many scales at once.

Keep going

Finished this one? 0 / 61 Labs done
NextThe Switchboard: Mixture of Experts Next on the “The transformer, opened up” ladder · rung 4 of 5

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs