Gradient Descent

Gradient descent is the optimization loop behind training: compute the gradient (the direction of steepest increase of the loss), then step the weights the opposite way by an amount set by the learning rate.

Worked example: in practice it is stochastic (SGD) — each step uses a small random batch instead of the whole dataset, trading a noisier gradient for far cheaper, more frequent updates. Gotcha: the learning rate dominates everything — too high and it diverges, too low and it crawls — which is why the learning-rate schedule and adaptive optimizers exist.