Positional Encoding
How a transformer is told the ORDER of tokens, since attention itself is order-blind.
Positional encoding injects information about each token’s position into its representation, because self-attention on its own is permutation-invariant — it would treat “dog bites man” and “man bites dog” identically without it.
Worked example: the original Transformer added fixed sinusoidal patterns to the embeddings; modern models often use rotary position embeddings (RoPE), which encode relative position by rotating query and key vectors. Gotcha: the positional scheme sets how well a model handles sequences longer than it trained on — some extrapolate poorly, which is why long-context variants re-scale or interpolate their position encodings.