Rotary position embedding,
explained.

Self-attention computes how much each token should attend to each other by comparing their query and key vectors — but it does this over an unordered set. Shuffle the words and, without extra information, the mechanism produces the same result. "Dog bites man" and "man bites dog" would be indistinguishable. So every transformer must inject position information somewhere.

The original transformer added a fixed sinusoidal position vector to each token's embedding. It works, but it is bolted on: position is mixed into the content, and what the model really cares about — how far apart two tokens are — has to be recovered indirectly. RoPE asks a sharper question: what if position lived in the geometry of the query and key comparison itself?

There are two ways to tell a model about order. Absolute encoding stamps each token with its index: this is position 5, this is position 6. Relative encoding cares instead about the offset between tokens: this token is 2 places before that one. For language, relative structure is usually what matters — "the adjective right before the noun" is a relative fact, true whether the phrase starts at word 3 or word 300.

Relative encoding also generalizes better. A model that has learned to handle offsets up to a few thousand does not care where in the sequence they occur, so it extends more gracefully to lengths it never saw in training. The challenge had always been building a relative scheme that stays cheap and slots naturally into the attention dot product. Rotation is that scheme.

Split each query and key vector into consecutive 2D pairs. For a token at position m, rotate every pair by an angle m·θ_i, where each pair i has its own frequency θ_i. In matrix form, apply a rotation R(mθ) to the query and R(nθ) to the key. Then take their dot product as usual. The magic is what that dot product simplifies to:

Because rotation matrices satisfy R(a)ᵀR(b) = R(b − a), the two absolute rotations collapse into a single rotation by the difference of positions. The attention score therefore depends only on n − m — the relative distance — even though you only ever rotated by absolute positions. Different pairs use different frequencies θ_i = base^−2i/d (base is usually 10000), so short-range and long-range relationships are each captured by their own rate of rotation. The runnable version below confirms both facts directly.

The whole result rests on one property of rotations: rotating by α then by β is the same as rotating by α + β. Angles simply add. So when a query rotated by mθ meets a key rotated by nθ, the comparison "un-rotates" one against the other and what remains is a rotation by (n − m)θ. Two consequences worth naming:

That last point matters practically: because rotation is norm-preserving, RoPE injects position without inflating or shrinking activations, keeping the attention scores well-behaved. Position becomes a pure change of direction, layered cleanly onto content that lives in magnitude and angle already.

The set of frequencies θ_i = base^−2i/d is doing real work. Early dimension pairs rotate fast (high frequency), capturing fine, short-range position; later pairs rotate slowly (low frequency), spanning long-range relationships. It is the same multi-scale idea as the original sinusoidal encoding, now expressed as rotation — a whole spectrum of clocks ticking at different rates so the model can read position at every scale.

This tunability is a large part of why RoPE became the standard as context windows grew from thousands to millions of tokens. Because position is a set of rotation frequencies, you can re-scale those frequencies to reach far beyond the training length — something an additive absolute embedding cannot do nearly as cleanly.

RoPE quietly became the default position encoding of the LLM era. LLaMA and its descendants, GPT-NeoX, PaLM, and most open models use it. It won by combining three things that had been hard to get together: it encodes relative position (what language needs), it slots directly into the existing attention dot product (no extra parameters, no architectural surgery), and it extends to longer contexts by rescaling its frequencies.

There is a deeper lesson in its elegance. Position had been treated as something you add to a representation; RoPE reframed it as something you do to the geometry of comparison. That change of perspective — from additive feature to rotational operation — is why a scheme this simple could quietly outlast the alternatives it replaced.