Cross-Attention

Cross-attention is attention between two different sequences: the queries come from one (say the decoder’s current output) and the keys and values from another (the encoder’s representation of the input). It is how an encoder-decoder model lets the output attend to the input.

Worked example: in translation, each decoder position uses cross-attention to look back at the encoded source sentence and pull the relevant words; self-attention, by contrast, attends within a single sequence. Gotcha: decoder-only LLMs have no separate encoder, so they use self-attention only — cross-attention shows up in encoder-decoder models and in multimodal models fusing, say, image features into text.