Softmax
Turns a vector of raw scores (logits) into a probability distribution that sums to 1.
Softmax exponentiates each score and normalizes, mapping arbitrary logits to probabilities. It sits at the output of a classifier or language model (over the next token) and inside attention (over the keys). The numerically stable form subtracts the max before exponentiating.
Worked example: logits [2, 1, 0] exponentiate to [7.39, 2.72, 1.0], sum 11.1, giving probabilities ≈ [0.66, 0.24, 0.09] — the gap between scores is amplified into a much larger gap between probabilities. Gotcha: exponentiating large logits overflows, so real implementations subtract the max first (softmax(x) = softmax(x − max x)) — mathematically identical, numerically safe. Skipping that is a classic from-scratch bug.