Adam,
explained.

Plain gradient descent has one knob — the learning rate — applied identically to every parameter. That is a problem, because parameters are not alike. Some receive large, frequent gradients; others receive tiny, rare ones. A step size big enough to move the sluggish parameters will make the sensitive ones explode; a step size safe for the sensitive ones leaves the sluggish ones barely moving. And the loss landscape makes it worse: real objectives have ravines — steep in one direction, nearly flat in another — where a single learning rate either bounces across the walls or crawls along the floor.

Two fixes had been invented separately. Momentum smooths and accelerates by averaging gradients over time. RMSProp gives each parameter its own effective learning rate, scaled by how big its gradients have recently been. Adam's contribution was to combine them cleanly — and to fix a subtle bias that both introduce at the start.

Instead of stepping along the raw gradient g, which is noisy and can zig-zag, momentum steps along an exponentially weighted average of recent gradients. Call it m — the first moment, an estimate of the gradient's mean. Each step nudges the old average a little toward the new gradient:

With β₁ = 0.9, roughly the last ten gradients dominate the average. The effect is a heavy ball rolling downhill: consistent directions accumulate and speed up, while noisy back-and-forth components cancel out. Ravines that make raw descent oscillate get smoothed into a steady roll along the valley floor.

Momentum fixes direction; it does nothing about scale. RMSProp adds a second running average — this time of the squared gradients. Call it v, the second moment, an estimate of each parameter's gradient variance:

Then it divides the step by √v. A parameter with consistently large gradients gets a large v and so a smaller effective step; a parameter with tiny gradients gets a small v and a larger effective step. Every parameter ends up with its own adaptive learning rate, automatically. With β₂ = 0.999, this variance is averaged over a much longer window than the momentum — it changes slowly and stably.

Put the two together — step in the direction of m, scaled by 1/√v — and you almost have Adam. Almost, because both averages start life at zero, and that creates a bias worth fixing.

Both moments are initialized at m = 0, v = 0. So on step 1, with a gradient g, the first moment is m = (1 − β₁)·g = 0.1·g — ten times too small. The estimate is biased toward zero, badly, for the first several steps. Adam cancels this exactly by dividing each moment by 1 − β^t, where t is the step number:

At t=1 the divisor 1−β₁ turns m = 0.1·g back into m̂ = g, undoing the bias precisely; as t grows, β^t → 0 and the correction fades to 1. The final line is the whole optimizer: move each parameter by the (corrected) average gradient, divided by the square root of its (corrected) average squared gradient.

One elegant property falls out of that division: Adam is roughly scale-invariant. Multiply every gradient by a constant and m̂ scales by that constant while √v̂ scales by the same amount — they cancel, and the step size barely changes. The update is governed by the direction and consistency of the gradients, not their raw magnitude, which is a big reason the default learning rate works across so many different problems.

The whole algorithm, per step t, per parameter: update m and v, bias-correct both, then step. Four hyperparameters govern it, and the paper's defaults are so robust they are rarely touched:

The reason those numbers travel so well is the scale invariance from section 04: because the update normalizes by the gradient's own size, the same α = 0.001 is a reasonable step whether gradients are large or small, whether the network is a tiny MLP or a giant transformer. That "just works out of the box" quality is exactly why Adam displaced hand-tuned SGD schedules for most practitioners.

Adam won because it removed a chore. Before it, getting a network to train meant babysitting a learning-rate schedule and often a per-layer tuning ritual. Adam's per-parameter adaptation absorbed most of that, converging fast and reliably on a huge range of architectures with the stock settings. For research velocity — try an idea, see if it trains, today — that mattered enormously.

It is not the end of the story. Plain Adam mishandles weight decay: folding L2 regularization into the gradient lets the adaptive 1/√v̂ scaling rescale the decay unevenly across parameters. AdamW fixes this by decoupling weight decay — applying it straight to the weights, outside the adaptive step — and is now the standard for training large models. Newer optimizers (LAMB for huge batches, Lion, Adafactor for memory) refine the idea further, but every one of them is a descendant of the same two-moments-plus-correction recipe this paper introduced.