Self-attention has a curious blind spot: it treats a sequence as a bag of tokens. The dot product between a query and a key doesn’t care whether two words are adjacent or a hundred apart, so a raw transformer literally cannot tell “dog bites man” from “man bites dog.” Something has to inject position. Rotary Position Embedding (RoPE) — used in LLaMA, GPT-NeoX, and most modern open models — does it with an idea that is as elegant as it is practical: rotate each token’s query and key vector by an angle proportional to its position. A token at position 3 gets rotated three times as far as a token at position 1. The magic falls out of basic geometry: the dot product of two vectors depends only on the angle between them, and after rotating a query by m·θ and a key by n·θ, the angle between them shifts by exactly (n − m)·θ. So the attention score ends up depending only on the relative distancen − m between the two tokens — never on their absolute positions. Slide two tokens along the sequence and watch their attention score stay fixed as long as the gap between them does.
attention is order-blind · rotate query/key by position·θ · dot product depends on relative offset (n−m)
Rotary Position Embedding: rotate each token’s query and key vector by an angle proportional to its position.
relative position
What actually matters for many relationships — how far apart two tokens are, not their absolute indices.
rotation invariance
A dot product depends only on the angle between two vectors, so rotating both by the same amount leaves it unchanged.
rope.js — rotate by position, score by distance
Ready
Two tokens — a query at position m and a key at position n — each rotated by its position. Move them along the sequence and watch the attention score (their dot product). Try shift both: the score won’t change, because only the gap n−m matters.
0
query pos m
3
key pos n
-0.34
score (offset 3)
How it works
The reason RoPE feels almost too clean is that it turns “where is this token?” into a rotation, and rotations interact with dot products in exactly the way attention needs. Take a query vector and a key vector living in a 2-D plane. RoPE rotates the query by an angle proportional to the query’s position — m·θ — and the key by an angle proportional to the key’s position — n·θ. Now recall the one fact that makes it all work: the dot product of two vectors equals |a||b|·cos(angle between them), so it is completely determined by the angle between them, not their orientation in the plane. Rotating the query by m·θ and the key by n·θ changes the angle between them by (n − m)·θ — the difference of the positions. Therefore the attention score is a function of n − m alone: it is translation-invariant. That single property is why RoPE is so effective. It means the model learns relationships in terms of relative distance (“the adjective two words before the noun”), which generalizes far better than pinning meaning to absolute slots; it needs no learned position table, so nothing has to be trained per-position; and there is no table that runs out at a fixed length. That is not the same as working at any length: gaps longer than any seen in training are still unfamiliar, so quality drops past the trained length unless the angles are rescaled (position interpolation, YaRN). Real models don’t use a single 2-D plane — they split the head dimension into many pairs and give each pair its own rotation frequency θ (high frequencies capture fine, nearby distinctions; low frequencies capture coarse, long-range ones), which is directly analogous to how sinusoidal encodings use many wavelengths. But every one of those planes obeys the same rule you can see here: rotate by position, and the score comes out depending only on the distance.
1
Attention is order-blind
A plain dot product between a query and a key ignores where the tokens sit in the sequence. Without extra information, the model can’t distinguish word orderings — position must be injected.
2
Rotate each vector by its position
RoPE rotates the query by an angle m·θ (m = its position) and the key by n·θ (n = its position). A token twice as far along is rotated twice as much.
3
The dot product sees only the gap
Because a dot product depends only on the angle between two vectors, rotating the query by m·θ and the key by n·θ changes that angle by (n−m)·θ. The score becomes a function of the relative distance n−m alone.
✓
Relative and table-free
The score is translation-invariant: shift both tokens and it doesn’t change. So the model learns relative relationships and needs no learned position table. Going past the trained length still takes rescaled angles. Real models do this across many rotation frequencies at once.
Problem
attention ignores order
RoPE
rotate q,k by position·θ
Score depends on
relative offset n−m
Bonus
no learned position table
The code
# RoPE in one 2-D plane (real models do many planes/frequencies)defrotate(v, angle):
c, s = cos(angle), sin(angle)
return [v[0]*c - v[1]*s, v[0]*s + v[1]*c]
q_m = rotate(q, m * theta) # query rotated by its position m
k_n = rotate(k, n * theta) # key rotated by its position n
score = dot(q_m, k_n) # = dot(q, rotate(k, (n - m)*theta))# depends ONLY on the relative offset (n - m) — translation-invariant
The picture: same gap, same score
Shifting both tokens by two positions turns both arrows by 60°, so the angle between them stays 110° and the dot product stays −0.34. The table is the whole score function for this plane: one number per offset, whatever the absolute positions. With a single 30° frequency it repeats every 12 positions, which is why real models use many frequencies.
Worked example: the lab’s numbers, row by row
The lab rotates by θ = 30° per position. The query starts at 0° and the key at 20°, both of length 1, so the score is just the cosine of the angle between them. Query angle = 30° × m; key angle = 20° + 30° × n.
m
n
Query angle
Key angle
Gap
Score
0
0
0°
20°
20°
0.94
0
1
0°
50°
50°
0.64
0
3
0°
110°
110°
−0.34
2
5
60°
170°
110°
−0.34
6
9
180°
290°
110°
−0.34
0
12 (past the lab’s 9)
0°
380°
380° = 20°
0.94
Check one row by coordinates. At m = 2 the query is (cos 60°, sin 60°) = (0.500, 0.866); at n = 5 the key is (cos 170°, sin 170°) = (−0.985, 0.174). The dot product is 0.500 × −0.985 + 0.866 × 0.174 = −0.342, the same as the cosine of 110°. The last row is the catch. One plane with one frequency cannot tell offset 12 from offset 0. Real models give each pair of dimensions its own frequency, θ_i = 10000^(−2i/d) in the original RoPE. For a head of 128 dimensions the fastest pair turns 1 radian per position and repeats every 6.28 positions; the slowest repeats only every 54,410 positions. Read together, the pairs give every offset in a normal context its own pattern.
Cost and trade-offs: position schemes compared
Here Time is the extra compute and Space is the extra learned parameters. All of them are small next to attention itself; the real differences are in what the model can generalise to.
Case
Time
Space
Why
Learned absolute table
One lookup and add per token
max_len × d parameters
Simple, but has no entry for any position past max_len.
Sinusoidal absolute
One add per token, at the input
None
A fixed formula mixed into the token embedding once. Relative distance has to be learned.
RoPE
Rotate q and k in every attention layer, O(d) per token
None
Relative position falls out of the dot product. Values are left alone.
ALiBi
Add −slope × distance to every attention score
None
No rotation: a straight penalty that grows with distance. BLOOM used it.
Learned relative bias (T5)
Add a learned bias per distance bucket to every score
A small table of bucket biases
Far distances share a bucket, so the model sees “far” rather than an exact gap.
RoPE past the trained length
Same
None
Gaps the model never saw in training give unfamiliar rotation patterns, and quality usually drops sharply. Position interpolation and YaRN rescale the angles to fit the trained range, usually with a short fine-tune.
That last row is the limit on longer sequences. A gap of 5 does look the same everywhere, so RoPE generalises across where a pattern sits. It does not, by itself, generalise to gaps longer than any it was trained on. That takes the angle rescaling above.
Where this shows up in production and interviews
Stretching a model’s context
Long-context versions of RoPE models are usually made by changing the angles, not the architecture: raise the base, or rescale with position interpolation or YaRN, then fine-tune briefly on long text. In Hugging Face model configs this appears as rope_theta and rope_scaling.
Keys are rotated before they are cached, so each cached key already carries its position. Generating one more token rotates only the new query and key. Nothing in the cache is touched again.
Images and video
Vision and multimodal models extend the idea to 2-D or 3-D by giving some dimension pairs the row position and others the column (or time) position. The same dot-product rule then gives relative position in each direction.
Interview angle
Common questions: how can relative position come from absolute rotations? (Only the difference of angles reaches the dot product.) Why rotate q and k but not v? (Position should change who attends to whom, not the content passed along.) What breaks past the training length, and how is it fixed?
Common mistakes with RoPE
Rotating the values too
Only q and k are rotated. Position should change who attends to whom; rotating v would scramble the content that gets passed along instead.
Pairing the wrong dimensions
RoPE rotates dimensions in pairs, and implementations differ: some pair neighbours (0 with 1), others pair the first half with the second (0 with d/2). Loading weights trained one way into code written the other way gives a model that runs and quietly produces worse attention.
Rotating cached keys twice
Keys are rotated once, at their own position, before they go into the KV cache. Rotating the whole cache again at every step shifts every cached key by the new position and breaks the relative distances.
Assuming it extrapolates for free
A gap of 5 looks the same at any position, but a gap longer than anything seen in training is still new to the model. Without rescaled angles (position interpolation, YaRN) quality usually drops past the trained length.
Using one frequency
With a single rotation speed, as in this lab, offset 12 lands on the same angle as offset 0 (30° × 12 = 360°). Real models spread pairs across many frequencies so that every offset in a normal context gets its own pattern.
Quick check
1. Why does a transformer need positional encoding at all?
Self-attention treats the sequence as an unordered set: the dot product between a query and key doesn’t depend on where the tokens are. So “dog bites man” and “man bites dog” look identical unless position is injected — which is what positional encodings like RoPE provide.
2. How does RoPE encode a token’s position?
RoPE rotates each token’s query and key by an angle equal to its position times a base frequency θ. A token at position 3 is rotated three times as far as one at position 1. (Learned position tables are a different, absolute approach.)
3. Why does the RoPE attention score depend only on the relative distance between tokens?
The dot product equals |q||k|·cos(angle between them). Rotating the query by m·θ and the key by n·θ changes the angle between them by exactly (n−m)·θ, so the score is a function of the relative offset n−m alone — it’s translation-invariant.
FAQ
What is RoPE (rotary position embedding)?
A way to give a transformer positional awareness by rotating each token’s query and key vectors by an angle proportional to the token’s position. Since the attention dot product depends only on the angle between vectors, rotating the query and key by their positions makes the score depend only on the relative distance between the two tokens. It’s used in many modern LLMs (LLaMA, GPT-NeoX, Mistral) because it encodes relative position naturally, needs no learned position table, and can be stretched to longer contexts by rescaling its angles (position interpolation, YaRN). Without that rescaling, quality still drops past the length it was trained on.
How is RoPE different from absolute positional encodings?
Absolute encodings add a position-dependent vector to each token, tagging its absolute index, and the model learns what the tags mean. RoPE instead bakes position into the query/key interaction by rotation, so the score sees the relative offset between tokens, not absolute positions. This generalizes better (many relationships are inherently relative) and needs no per-position learned parameters, so there is no table that runs out at a fixed length. Past the trained length the model still degrades unless the angles are rescaled (position interpolation, YaRN).
Why does encoding relative position help language models?
Because most linguistic structure is about how far apart things are, not their absolute index. Subject–verb agreement, an adjective modifying a nearby noun, a pronoun referring to a recent entity — these depend on relative distance and are the same whether the phrase is early or late in the text. A model encoding relative position learns such a pattern once and applies it everywhere. That is not the same as handling any length: gaps longer than any seen in training are still unfamiliar, so quality drops past the trained length unless the angles are rescaled (position interpolation, YaRN).
How does RoPE work across many dimensions, not just a 2-D plane?
A real head has a query/key dimension of 64 or 128, and RoPE splits it into coordinate pairs, treating each as a 2-D plane with its own rotation frequency θ. Some pairs rotate quickly with position (high frequency), capturing fine short-range distinctions; others rotate slowly (low frequency), capturing long-range relationships — analogous to the spectrum of wavelengths in sinusoidal encodings. Every pair still obeys the same rule (its score contribution depends only on the relative offset), so the head encodes relative position across many scales at once.