Cross-Entropy Loss
The training loss that measures how surprised the model was by the correct answer.
Cross-entropy is the negative log-probability the model assigned to the true class, −log(p[target]). It’s near 0 when the model is confidently right and explodes when it’s confidently wrong — the signal that trains almost every classifier and language model.
Worked example: if the model gives the correct token probability 0.9, the loss is −log(0.9) ≈ 0.11; if it gives 0.1, the loss is −log(0.1) ≈ 2.30 — over 20× worse for the same one prediction. Gotcha: because the penalty goes to infinity as the true-class probability approaches 0, a single confidently-wrong token dominates the average loss — which is why models learn hard to never be certain and wrong, and why label noise is so damaging.