Gradient Descent
also: SGD · stochastic gradient descent
Nudge each weight a little in the direction that most reduces the loss; repeat until it stops improving.
Gradient descent is the optimization loop behind training: compute the gradient (the direction of steepest increase of the loss), then step the weights the opposite way by an amount set by the learning rate.
Worked example: in practice it is stochastic (SGD) — each step uses a small random batch instead of the whole dataset, trading a noisier gradient for far cheaper, more frequent updates. Gotcha: the learning rate dominates everything — too high and it diverges, too low and it crawls — which is why the learning-rate schedule and adaptive optimizers exist.