Rather than adding a separate learned position embedding to each token (which struggles to generalize past the lengths seen in training), RoPE rotates each query and key vector by an angle that depends on its position, so the dot product between two tokens naturally encodes their relative distance. This relative-position property is a big part of why RoPE-based models extend more gracefully to longer contexts than absolute position embeddings.
Worked example: rotary position embedding encodes token position by ROTATING each query/key vector by an angle proportional to its position, so attention naturally depends on relative distance rather than absolute index. Gotcha: because it is relative, RoPE extrapolates to longer contexts better than learned absolute embeddings — and its rotation frequencies can be interpolated/scaled (NTK, YaRN) to extend a model’s context window at inference without full retraining, a common way long-context variants are made.