RoPE (Rotary Position Embedding)
also: rotary position embedding
Encoding a token’s position by rotating its query/key vectors by an angle proportional to position, instead of adding a learned position vector.
Rather than adding a separate learned position embedding to each token (which struggles to generalize past the lengths seen in training), RoPE rotates each query and key vector by an angle that depends on its position, so the dot product between two tokens naturally encodes their relative distance. This relative-position property is a big part of why RoPE-based models extend more gracefully to longer contexts than absolute position embeddings.