Adam,
explained.

Rung 2 improves conditioning by changing the network — rescaling activations so the surface the optimiser walks is better behaved. Adam leaves the network alone and changes the walk. Plain SGD applies one global learning rate to every parameter, which assumes every parameter wants a step of roughly the same size. In a deep network they demonstrably do not: a rarely-activated embedding row and a hot attention projection have gradients that differ by orders of magnitude.

The change is to keep two running averages per parameter and divide one by the other. The first moment is an exponential average of the gradient — momentum, which carries you through flat regions and damps oscillation. The second moment is an exponential average of the squared gradient — the RMSProp term, which measures how large this parameter's gradients have recently been. Dividing the first by the root of the second gives a step whose size is roughly scale-free: a parameter with consistently tiny gradients still moves.

Stated in the ladder's units: rung 2 added two parameters per feature and bought a bigger learning rate. Adam adds two optimiser states per parameter — so an Adam run holds roughly three times the model's weights in memory, which is the single largest reason training a model costs more memory than serving it — and buys a per-parameter effective learning rate. It also introduces the correction that most reimplementations get wrong: both averages start at zero, so early in training both are biased toward zero, and the bias correction divides each by 1−β^^t to undo it. Skip it and the first few hundred steps take wildly wrong-sized steps. The famous defaults — β₁ 0.9, β₂ 0.999, ε 1e−8 — are from this paper and are still the usual starting point, although large-language-model training commonly lowers β₂ to roughly 0.95–0.98 and adds learning-rate warmup.

Plain gradient descent has one knob — the learning rate — applied identically to every parameter. That is a problem, because parameters are not alike. Some receive large, frequent gradients; others receive tiny, rare ones. A step size big enough to move the sluggish parameters will make the sensitive ones explode; a step size safe for the sensitive ones leaves the sluggish ones barely moving. And the loss landscape makes it worse: real objectives have ravines — steep in one direction, nearly flat in another — where a single learning rate either bounces across the walls or crawls along the floor.

Two fixes had been invented separately. Momentum smooths and accelerates by averaging gradients over time. RMSProp gives each parameter its own effective learning rate, scaled by how big its gradients have recently been. Adam's contribution was to combine them cleanly — and to fix a subtle bias that both introduce at the start.

Instead of stepping along the raw gradient g, which is noisy and can zig-zag, momentum steps along an exponentially weighted average of recent gradients. Call it m — the first moment, an estimate of the gradient's mean. Each step nudges the old average a little toward the new gradient:

With β₁ = 0.9, roughly the last ten gradients dominate the average. The effect is a heavy ball rolling downhill: consistent directions accumulate and speed up, while noisy back-and-forth components cancel out. Ravines that make raw descent oscillate get smoothed into a steady roll along the valley floor.

Momentum fixes direction; it does nothing about scale. RMSProp adds a second running average — this time of the squared gradients. Call it v, the second moment, an estimate of each parameter's gradient variance:

Then it divides the step by √v. A parameter with consistently large gradients gets a large v and so a smaller effective step; a parameter with tiny gradients gets a small v and a larger effective step. Every parameter ends up with its own adaptive learning rate, automatically. With β₂ = 0.999, this variance is averaged over a much longer window than the momentum — it changes slowly and stably.

Put the two together — step in the direction of m, scaled by 1/√v — and you almost have Adam. Almost, because both averages start life at zero, and that creates a bias worth fixing.

Both moments are initialized at m = 0, v = 0. So on step 1, with a gradient g, the first moment is m = (1 − β₁)·g = 0.1·g — ten times too small. The estimate is biased toward zero, badly, for the first several steps. Adam cancels this exactly by dividing each moment by 1 − β^^t, where t is the step number:

At t=1 the divisor 1−β₁ turns m = 0.1·g back into m̂ = g, undoing the bias precisely; as t grows, β^^t → 0 and the correction fades to 1. The final line is the whole optimizer: move each parameter by the (corrected) average gradient, divided by the square root of its (corrected) average squared gradient.

One elegant property falls out of that division: Adam is roughly scale-invariant. Multiply every gradient by a constant and m̂ scales by that constant while √v̂ scales by the same amount — they cancel, and the step size barely changes. The update is governed by the direction and consistency of the gradients, not their raw magnitude, which is a big reason the default learning rate works across so many different problems.

Kingma and Ba present Adam as a first-order method for stochastic objectives with many parameters, where the noise can come from mini-batch sampling or from dropout and the objective may drift over time. Beyond the moment updates written out on this page, the paper specifies:

1. Initialization. Choose α and β₁, β₂ in [0, 1); set m₀ = v₀ = 0 and t = 0. All operations are elementwise, so the optimizer state is two numbers per parameter.
2. A cheaper ordering of the same step. Fold both bias corrections into one scalar, α_t = α·√(1 − β₂^^t)/(1 − β₁^^t), then update θ_t = θ_t−1 − α_t·m_t/(√v_t + ε̂). This avoids building m̂ and v̂ as separate vectors.
3. α as a trust region. Ignoring ε, the effective step is Δ_t = α·m̂_t/√v̂_t, and its size is approximately bounded by α (the exact bounds are written out with the equations). The authors suggest choosing α from how far the optimum plausibly sits from the starting point, since α caps how far one step can travel.
4. Automatic annealing. The paper calls m̂/√v̂ a signal-to-noise ratio. When successive gradients disagree in sign, the average m̂ shrinks while √v̂ does not, so the effective step falls toward zero, as it typically does near an optimum.
5. Analysis in the online convex setting. The regret bound assumes convex losses, bounded gradients and iterates, a step size decaying as α/√t, a first-moment coefficient decaying as β₁λ^^t−1 with λ just below 1, and β₁²/√β₂ < 1, and gives regret growing as √T. In practice Adam is run with constant β₁, and the proof was later shown to contain an error.

Where it sits among adaptive methods

AdaGrad divides each step by the root of the sum of all past squared gradients. The paper shows Adam reduces to it with β₁ = 0, β₂ infinitesimally close to 1 and a step α/√t, but only with bias correction: without it, β₂ → 1 would give infinite bias and infinite steps. RMSProp with momentum is the closest relative, but it applies momentum to the already-rescaled gradient instead of averaging the raw gradient, and it has no bias correction, which matters most when β₂ is near 1, the setting sparse gradients need. The authors also point out that v̂ approximates the diagonal of the Fisher information, so Adam behaves like a conservative form of natural-gradient descent that preconditions with the square root of that diagonal.

AdaMax and parameter averaging

Replacing the L2 norm in the denominator with an Lp norm becomes numerically unstable for large p, but the limit p → ∞ is simple and stable: track an exponentially weighted infinity norm u_t = max(β₂·u_t−1, |g_t|), which needs no bias correction, and step by α/(1 − β₁^^t) times m_t/u_t. Its suggested defaults are α = 0.002, β₁ = 0.9 and β₂ = 0.999. The paper also proposes keeping an exponential moving average of the parameters themselves, bias-corrected the same way, because the last iterate of a stochastic method is noisy.

• Δt — the change applied to one parameter at step t, with ε ignored.
• (1 − β₁)/√(1 − β₂) — the worst case, which the paper says occurs only under the most severe sparsity: a gradient that was zero at every earlier step and is nonzero now. With the defaults it is 0.1/√0.001 ≈ 3.16, reached once the bias corrections have faded.
• |m̂/√v̂| ≤ 1 — the common case, since |E[g]|/√E[g²] ≤ 1; steps are close to α when gradients agree in sign and smaller when they do not.
• ut — an exponentially decaying maximum of past |g|, the limit of the Lp version as p → ∞. It starts at 0 but jumps to |g₁| immediately, so unlike v it is not biased toward zero and needs no correction; the paper gives AdaMax the simpler bound |Δ_t| ≤ α.

At the extremes: with (1 − β₁) = √(1 − β₂), |m̂/√v̂| stays below 1 and no step exceeds α. With β₁ = 0, β₂ → 1 and a step size of α/√t, the update becomes AdaGrad's α·g_t/√(Σ_i≤t g_i²), the method Adam was designed to generalize.

Each comparison used the same model for every optimizer, with hyperparameters such as the learning rate and momentum chosen by a dense grid search and the best setting reported. The results are training-cost curves rather than accuracy tables, so the findings below are the authors' readings of those curves.

Two findings cut against the simplest story. On the CNN, the second-moment estimate v̂ fell to nearly zero within a few epochs and was dominated by ε, so the authors conclude it described the cost surface poorly there and that most of the speed-up came from the first moment. What Adam still offered was a separate effective step size per layer, without hand-picking a smaller rate for the convolutional layers as is common with SGD. SFO, the quasi-Newton competitor, was 5 to 10 times slower per iteration and needed memory growing with the number of mini-batches. In the bias-correction ablation the best results combined a slow second-moment decay with the correction, and across the whole grid Adam did as well as or better than the uncorrected version.

The paper offers a convergence argument for convex objectives, and that argument turned out to be wrong. Reddi and colleagues exhibited a simple convex problem in 2018 on which Adam does not converge to the optimum, because the exponential average of squared gradients can forget a rare but informative large gradient; their AMSGrad variant keeps a running maximum instead. This is the clearest case on the ladder of a method that is overwhelmingly useful and whose original proof does not hold.

A second defect is not admitted because it was not noticed: the paper folds L2 regularisation into the gradient, which means the penalty is then divided by the second-moment term along with everything else, so parameters with large gradients are effectively regularised less. Loshchilov and Hutter separated the two — decoupling weight decay from the adaptive scaling — and AdamW, not Adam, is what actually trains modern models.

How it held up: as the default, almost completely. Essentially every transformer on this site was trained with Adam or AdamW. Two things remain genuinely contested. The first is generalisation: on image classification, well-tuned SGD with momentum often matches or beats Adam on held-out accuracy, and the reason is not settled. The second is whether the per-parameter scalar is the right object at all — rung 7 of the scaling ladder is the first serious challenger to get traction, and it treats a weight matrix rather than each weight separately.

The whole algorithm, per step t, per parameter: update m and v, bias-correct both, then step. Four hyperparameters govern it, and the paper's defaults are so robust they are rarely touched:

The reason those numbers travel so well is the scale invariance from section 04: because the update normalizes by the gradient's own size, the same α = 0.001 is a reasonable starting step whether gradients are large or small, though it is not a good value for giant transformers, where the peak rate is usually far smaller and warmed up. That "just works out of the box" quality is exactly why Adam displaced hand-tuned SGD schedules for most practitioners.

Adam won because it removed a chore. Before it, getting a network to train meant babysitting a learning-rate schedule and often a per-layer tuning ritual. Adam's per-parameter adaptation absorbed most of that, converging fast and reliably on a huge range of architectures with the stock settings. For research velocity — try an idea, see if it trains, today — that mattered enormously.

It is not the end of the story. Plain Adam mishandles weight decay: folding L2 regularization into the gradient lets the adaptive 1/√v̂ scaling rescale the decay unevenly across parameters. AdamW fixes this by decoupling weight decay — applying it straight to the weights, outside the adaptive step — and is now the standard for training large models. Newer optimizers (LAMB for huge batches, Lion, Adafactor for memory) refine the idea further, though not all of them keep the recipe: LAMB is Adam with a per-layer trust ratio, while Adafactor factors the second moment and Lion tracks only a momentum and takes a sign.

Work each one out before you open it — a number, a prediction or a reason, not a definition. Producing the answer is what makes it stick. Your progress is kept on this device.

Seven papers that turned "deeper is better in theory" into a network that actually converges. Every later architecture paper assumes all of them.

1. 1Dropout2014
2. 2Batch Normalization2015
3. 3Adam2015
4. 4ResNet2016
5. 5LayerNorm & RMSNorm2016
6. 6mixup2018
7. 7The Lottery Ticket Hypothesis2019

All 110 papers, in 19 ladders →