Self-attention has a curious blind spot: it treats a sequence as a bag of tokens. The dot product between a query and a key doesn’t care whether two words are adjacent or a hundred apart, so a raw transformer literally cannot tell “dog bites man” from “man bites dog.” Something has to inject position. Rotary Position Embedding (RoPE) — used in LLaMA, GPT-NeoX, and most modern open models — does it with an idea that is as elegant as it is practical: rotate each token’s query and key vector by an angle proportional to its position. A token at position 3 gets rotated three times as far as a token at position 1. The magic falls out of basic geometry: the dot product of two vectors depends only on the angle between them, and after rotating a query by m·θ and a key by n·θ, the angle between them shifts by exactly (n − m)·θ. So the attention score ends up depending only on the relative distancen − m between the two tokens — never on their absolute positions. Slide two tokens along the sequence and watch their attention score stay fixed as long as the gap between them does.
attention is order-blind · rotate query/key by position·θ · dot product depends on relative offset (n−m)
positional encoding
Extra information injected into a transformer so it knows the order of tokens — attention alone is order-blind.
Rotary Position Embedding: rotate each token’s query and key vector by an angle proportional to its position.
relative position
What actually matters for many relationships — how far apart two tokens are, not their absolute indices.
rotation invariance
A dot product depends only on the angle between two vectors, so rotating both by the same amount leaves it unchanged.
rope.js — rotate by position, score by distance
Ready
Two tokens — a query at position m and a key at position n — each rotated by its position. Move them along the sequence and watch the attention score (their dot product). Try shift both: the score won’t change, because only the gap n−m matters.
0
query pos m
3
key pos n
-0.34
score (offset 3)
How it works
The reason RoPE feels almost too clean is that it turns “where is this token?” into a rotation, and rotations interact with dot products in exactly the way attention needs. Take a query vector and a key vector living in a 2-D plane. RoPE rotates the query by an angle proportional to the query’s position — m·θ — and the key by an angle proportional to the key’s position — n·θ. Now recall the one fact that makes it all work: the dot product of two vectors equals |a||b|·cos(angle between them), so it is completely determined by the angle between them, not their orientation in the plane. Rotating the query by m·θ and the key by n·θ changes the angle between them by (n − m)·θ — the difference of the positions. Therefore the attention score is a function of n − m alone: it is translation-invariant. That single property is why RoPE is so effective. It means the model learns relationships in terms of relative distance (“the adjective two words before the noun”), which generalizes far better than pinning meaning to absolute slots; it needs no learned position table, so nothing has to be trained per-position; and it degrades gracefully when you run the model on sequences longer than it was trained on, because a gap of 5 looks the same everywhere. Real models don’t use a single 2-D plane — they split the head dimension into many pairs and give each pair its own rotation frequency θ (high frequencies capture fine, nearby distinctions; low frequencies capture coarse, long-range ones), which is directly analogous to how sinusoidal encodings use many wavelengths. But every one of those planes obeys the same rule you can see here: rotate by position, and the score comes out depending only on the distance.
1
Attention is order-blind
A plain dot product between a query and a key ignores where the tokens sit in the sequence. Without extra information, the model can’t distinguish word orderings — position must be injected.
2
Rotate each vector by its position
RoPE rotates the query by an angle m·θ (m = its position) and the key by n·θ (n = its position). A token twice as far along is rotated twice as much.
3
The dot product sees only the gap
Because a dot product depends only on the angle between two vectors, rotating the query by m·θ and the key by n·θ changes that angle by (n−m)·θ. The score becomes a function of the relative distance n−m alone.
✓
Relative, table-free, extrapolates
The score is translation-invariant: shift both tokens and it doesn’t change. So the model learns relative relationships, needs no learned position table, and handles longer sequences gracefully. Real models do this across many rotation frequencies at once.
Problem
attention ignores order
RoPE
rotate q,k by position·θ
Score depends on
relative offset n−m
Bonus
no learned position table
The code
# RoPE in one 2-D plane (real models do many planes/frequencies)defrotate(v, angle):
c, s = cos(angle), sin(angle)
return [v[0]*c - v[1]*s, v[0]*s + v[1]*c]
q_m = rotate(q, m * theta) # query rotated by its position m
k_n = rotate(k, n * theta) # key rotated by its position n
score = dot(q_m, k_n) # = dot(q, rotate(k, (n - m)*theta))# depends ONLY on the relative offset (n - m) — translation-invariant
Quick check
1. Why does a transformer need positional encoding at all?
Self-attention treats the sequence as an unordered set: the dot product between a query and key doesn’t depend on where the tokens are. So “dog bites man” and “man bites dog” look identical unless position is injected — which is what positional encodings like RoPE provide.
2. How does RoPE encode a token’s position?
RoPE rotates each token’s query and key by an angle equal to its position times a base frequency θ. A token at position 3 is rotated three times as far as one at position 1. (Learned position tables are a different, absolute approach.)
3. Why does the RoPE attention score depend only on the relative distance between tokens?
The dot product equals |q||k|·cos(angle between them). Rotating the query by m·θ and the key by n·θ changes the angle between them by exactly (n−m)·θ, so the score is a function of the relative offset n−m alone — it’s translation-invariant.
FAQ
What is RoPE (rotary position embedding)?
A way to give a transformer positional awareness by rotating each token’s query and key vectors by an angle proportional to the token’s position. Since the attention dot product depends only on the angle between vectors, rotating the query and key by their positions makes the score depend only on the relative distance between the two tokens. It’s used in many modern LLMs (LLaMA, GPT-NeoX, Mistral) because it encodes relative position naturally, needs no learned position table, and extrapolates reasonably to longer sequences.
How is RoPE different from absolute positional encodings?
Absolute encodings add a position-dependent vector to each token, tagging its absolute index, and the model learns what the tags mean. RoPE instead bakes position into the query/key interaction by rotation, so the score sees the relative offset between tokens, not absolute positions. This generalizes better (many relationships are inherently relative), needs no per-position learned parameters, and behaves more gracefully on sequences longer than those seen in training.
Why does encoding relative position help language models?
Because most linguistic structure is about how far apart things are, not their absolute index. Subject–verb agreement, an adjective modifying a nearby noun, a pronoun referring to a recent entity — these depend on relative distance and are the same whether the phrase is early or late in the text. A model encoding relative position learns such a pattern once and applies it everywhere, and is more robust to length: a gap of five looks the same wherever it occurs, helping on inputs longer than training data.
How does RoPE work across many dimensions, not just a 2-D plane?
A real head has a query/key dimension of 64 or 128, and RoPE splits it into coordinate pairs, treating each as a 2-D plane with its own rotation frequency θ. Some pairs rotate quickly with position (high frequency), capturing fine short-range distinctions; others rotate slowly (low frequency), capturing long-range relationships — analogous to the spectrum of wavelengths in sinusoidal encodings. Every pair still obeys the same rule (its score contribution depends only on the relative offset), so the head encodes relative position across many scales at once.