Loss Function
also: objective function
The single number training tries to minimize — it measures how wrong the model’s predictions are.
A loss function (objective) is the formula that measures how wrong a model’s predictions are on the training data, producing a single number that training works to minimize via gradient descent. The choice of loss defines what “good” means for the model.
Worked example: classification uses cross-entropy (heavily penalizes confident wrong answers); regression uses mean squared error (penalizes large misses quadratically); language models use next-token cross-entropy. Gotcha: the loss is a PROXY for what you actually care about — optimizing it perfectly can still miss the real goal if it is misspecified (a model minimizing average loss may fail the rare cases that matter), which is why metric choice, class weighting, and separate eval metrics beyond the training loss matter.