Rotary position embedding,
explained.

Rung 2 supplies position by adding a position vector to a token's embedding before attention ever happens. That works, and it has a structural oddity: the attention score between two tokens then depends on their two absolute positions, and any sense of "these are three apart" has to be reconstructed by the network from that. What a language model overwhelmingly wants is relative distance — the word before this one matters because it is before this one, not because it is word 4,096.

The change is to stop adding and start rotating. Treat each pair of dimensions in the query and key vectors as a point in a plane and rotate it by an angle proportional to the token's position. Then the dot product of a query at position m with a key at position n is, by the algebra of rotations, a function of the rotation m − n: relative position falls out of the arithmetic instead of being learned from absolute inputs. Different dimension pairs are rotated at different rates — a geometric spread of frequencies — so some pairs track fine local offsets and others track position at the scale of the whole sequence.

Stated in the ladder's units: sinusoidal encodings add a vector and change the values that flow through the network; RoPE modifies only the queries and keys, leaves the values untouched, adds no learned parameters at all, and the paper proves an upper bound on its attention score that decays on average with distance, so faraway tokens tend to interact less. The claim that made it spread beyond elegance is about extrapolation — a scheme defined by relative offsets has no built-in largest position, so in principle it might keep working past the length it was trained on — a hope the paper does not test, and one that plain RoPE did not meet in practice.

Self-attention computes how much each token should attend to each other by comparing their query and key vectors — but it does this over an unordered set. Shuffle the words and, without extra information, the mechanism produces the same result. "Dog bites man" and "man bites dog" would be indistinguishable. So every transformer must inject position information somewhere.

The original transformer added a fixed sinusoidal position vector to each token's embedding. It works, but it is bolted on: position is mixed into the content, and what the model really cares about — how far apart two tokens are — has to be recovered indirectly. RoPE asks a sharper question: what if position lived in the geometry of the query and key comparison itself?

There are two ways to tell a model about order. Absolute encoding stamps each token with its index: this is position 5, this is position 6. Relative encoding cares instead about the offset between tokens: this token is 2 places before that one. For language, relative structure is usually what matters — "the adjective right before the noun" is a relative fact, true whether the phrase starts at word 3 or word 300.

Relative encoding also generalizes better. A model that has learned to handle offsets up to a few thousand does not care where in the sequence they occur, so it was hoped to extend more gracefully to lengths it never saw in training (later work found plain RoPE needs help to do this). The challenge had always been building a relative scheme that stays cheap and slots naturally into the attention dot product. Rotation is that scheme.

Split each query and key vector into consecutive 2D pairs. For a token at position m, rotate every pair by an angle m·θ_i, where each pair i has its own frequency θ_i. In matrix form, apply a rotation R(mθ) to the query and R(nθ) to the key. Then take their dot product as usual. The magic is what that dot product simplifies to:

Because rotation matrices satisfy R(a)ᵀR(b) = R(b − a), the two absolute rotations collapse into a single rotation by the difference of positions. The attention score therefore depends only on n − m — the relative distance — even though you only ever rotated by absolute positions. Different pairs use different frequencies θ_i = base^^−2i/d (base is usually 10000), so short-range and long-range relationships are each captured by their own rate of rotation. The runnable version below confirms both facts directly.

The whole result rests on one property of rotations: rotating by α then by β is the same as rotating by α + β. Angles simply add. So when a query rotated by mθ meets a key rotated by nθ, the comparison "un-rotates" one against the other and what remains is a rotation by (n − m)θ. Two consequences worth naming:

That last point matters practically: because rotation is norm-preserving, RoPE injects position without inflating or shrinking activations, keeping the attention scores well-behaved. Position becomes a pure change of direction, layered cleanly onto content that lives in magnitude and angle already.

RoFormer is a Transformer in which RoPE replaces the absolute position embeddings. Everything else, from the projections to the softmax, is unchanged, and RoPE adds no trained parameters.

1. Project without positions. Token embeddings enter the network with no position vector added. Each attention layer computes queries and keys as usual, q = W_qx_m and k = W_kx_n.
2. Pair the dimensions. A head of dimension d is split into d/2 two-dimensional pairs. The paper pairs neighbouring coordinates, (x₁, x₂), (x₃, x₄) and so on, as GPT-J's code does. GPT-NeoX and the Hugging Face LLaMA code instead pair coordinate i with coordinate i + d/2, the version usually called rotate_half. The two are the same method up to a fixed reordering of coordinates, but a checkpoint trained with one only runs correctly under the other if the rows of W_q and W_k are permuted to match.
3. Give each pair a frequency. Pair j turns at θ_j = 10000^^{−2(j−1)/d} radians per token, a geometric spread from 1 radian per token down to roughly one ten-thousandth.
4. Rotate by position. In the query at position m, pair j is rotated by mθ_j; in the key at position n, by nθ_j. In code this is two element-wise products with precomputed cosine and sine tables plus a swap-and-negate of coordinates: O(d) work per token, and no n × n bias matrix is ever built.
5. Attend as usual, values untouched. The dot product of a rotated query and key depends on position only through n − m, so every score sees relative position directly. Values are not rotated, so the content being mixed carries no position signal of its own.
6. Repeat in every layer. An added embedding enters once at the bottom and has to survive the residual stream. The rotation is applied afresh to the queries and keys of every attention layer.

Two properties the paper proves

Long-term decay. The authors write the score as a sum of rotating complex terms and apply Abel summation. That bounds its size by a quantity whose average shrinks as |m − n| grows, given the 10000 frequency spread. It is a bound, not a promise that every score falls with distance: real scores oscillate underneath it.

Linear-attention compatibility. RoPE acts on each query and key separately, not on the n × n score matrix, so it can be added to kernelized linear attention such as the Performer. The paper rotates the feature-mapped queries and keys in the numerator and leaves the normalizing denominator unrotated, which avoids any risk of dividing by zero. The cost, which the authors acknowledge, is that the weights on the values are no longer strictly normalized.

How the paper tested it

The experiments swap RoPE into existing recipes instead of building new models. They cover a Transformer-base translation model on WMT 2014 English-German, a BERT-style masked language model pre-trained on BookCorpus and English Wikipedia and then fine-tuned on GLUE, a Performer language model on enwik8, and Chinese RoFormer models pre-trained with maximum lengths up to 1,024 tokens for a long-document task.

• x — a projected query or key of dimension d. The same function is applied to both, with that token's own position.
• ⊙ — element-wise product with tables that depend only on position, so they are computed once and reused for every layer and head.
• (−x₂, x₁, …) — each pair swapped with one sign flipped, which is what turns two multiplications into a 2-D rotation.
• θj — the rotation rate of pair j, in radians per token.
• q̃j, k̃j — pair j read as one complex number; * is the complex conjugate and i the imaginary unit.
• m − n — the only place positions appear in the score.

At the extremes: when m = n every phase is zero and the score is the plain dot product. The fastest pair completes a full turn about every 6 tokens, so for large offsets its phase wraps around and carries only local information. The slowest pair barely turns across a training context, so it behaves almost like a position-free channel. Raising the base above 10000 slows every pair down, which is the lever later long-context models pull.

Every experiment holds a published recipe fixed and changes only the position scheme, so each comparison is RoPE against whatever that recipe used before, usually learned or sinusoidal absolute positions.

Read honestly, the translation gain is small, 0.2 BLEU, and GLUE is mixed. The most distinctive results are the training curves: masked language modeling and the linear-attention Performer both converged faster with RoPE, which supports the claim that relative position is easier to learn when it is built into the score.

The long-document result needs one caveat. The gain at 1,024 tokens came from a model trained at that length reading more of each document, not from running a model beyond the length it was trained on. The paper does not test extrapolation, and when later work did, unmodified RoPE degraded badly past its training length.

The authors are explicit about what they could not explain. They prove the relative-position property and the long-term decay bound, but say they lack a thorough account of why RoPE converges faster than other position encodings, or why it does better on long texts when its decay property resembles that of existing schemes.

The extrapolation claim is the part that did not survive contact. In practice, a model trained at one context length and run at several times that length with RoPE unchanged degrades badly — the rotations reach angles the model has never seen, and quality falls off a cliff rather than tapering. This is the clearest case on the ladder of a property that is true of the encoding and false of the trained model.

What followed is a family of patches, and they are worth naming because any long-context model you use today contains one: interpolating positions so a longer sequence is squeezed into the trained angular range, adjusting the frequency spread so high-frequency dimensions are treated differently from low ones, and the combinations of the two that became standard practice for extending a released model's context after training. None of these are in the original paper; all of them are RoPE plus an admission.

How it held up: as the default. Rotary positions are what the open-weight models use and what most frontier architectures are assumed to use, and the alternative that competed most seriously — biasing attention scores linearly with distance — lost on quality rather than on principle. The honest summary is that RoPE won the question it was built for (represent relative position cleanly and cheaply) and lost the question it was marketed on (extend context for free), and that both halves are why the context ladder exists as separate engineering rather than a solved corollary.

RoPE quietly became the default position encoding of the LLM era. LLaMA and its descendants, GPT-NeoX, PaLM, and most open models use it. It won by combining three things that had been hard to get together: it encodes relative position (what language needs), it slots directly into the existing attention dot product (no extra parameters, no architectural surgery), and it extends to longer contexts by rescaling its frequencies.

There is a deeper lesson in its elegance. Position had been treated as something you add to a representation; RoPE reframed it as something you do to the geometry of comparison. That change of perspective — from additive feature to rotational operation — is why a scheme this simple could quietly outlast the alternatives it replaced.

RoPE — Rotary Position Embedding — is how almost every modern LLM tells tokens apart by position, and it quietly solved problems the original approaches had. Rather than adding a position vector to each token (absolute encoding) or bolting on learned position tables, RoPE rotates each token's query and key vectors by an angle proportional to its position. When two tokens interact through attention, the math works out so that only their relative distance affects the result.

That relative property is the win. Attention naturally cares about "how far apart are these tokens," not their absolute indices, and RoPE bakes that in elegantly — no extra parameters, applied directly inside the attention dot product. It gives clean relative-position behavior while being simple and efficient to compute.

RoPE also turned out to be friendly to context extension. Because position is encoded as rotation frequencies, you can stretch or interpolate those frequencies to make a model trained on short contexts handle longer ones — techniques like position interpolation and NTK/YaRN scaling all build on RoPE, and they are a big part of how models grew from a few thousand to hundreds of thousands of tokens of context.

It matters because it is now the default: RoPE (and its scaled variants) is in most open LLMs, and long-context work overwhelmingly assumes it. Understanding rotary embeddings is understanding how today's models represent "where" a token is — and how they get extended to read far more of it.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

The architecture itself: what it replaced, how position is encoded once recurrence is gone, and what it turns out to have learned inside.

1. 1Seq2Seq2014
2. 2Attention Is All You Need2017
3. 3RoPE2021
4. 4Concept Circuits2026

All 110 papers, in 19 ladders →