Training a neural network is, underneath all the vocabulary, rolling downhill: you have a loss surface and you take steps in the direction that lowers the loss. The catch is that real loss surfaces are ill-conditioned — far steeper in some directions than others, like a long narrow ravine — and on that terrain the exact update rule you use, the optimizer, makes an enormous difference to how fast you reach the bottom. Plain stochastic gradient descent (SGD) just steps opposite the gradient; in a ravine the gradient points mostly across the valley, so SGD bounces from wall to wall and inches forward along the floor. Momentum accumulates a velocity — steps in a consistent direction reinforce, and the back-and-forth oscillations cancel — so it builds speed along the valley and damps the bouncing. Adam goes further: it keeps a running estimate of each direction’s gradient magnitude and rescales the step per direction, so it takes big steps where the surface is shallow and small ones where it’s steep, heading almost straight for the minimum. Drop all three at the same point and watch their paths — and their losses — pull apart.
gradient descent on an ill-conditioned ravine · SGD zigzags · momentum accelerates · Adam adapts per-direction
loss surface
The landscape of loss values over the parameters. Training walks downhill on it; the minimum is the goal.
SGD
Step opposite the gradient, scaled by a learning rate. Simple, but zigzags on ill-conditioned surfaces.
momentum
Accumulate a velocity across steps: consistent directions build up, oscillations cancel — faster along a ravine.
Adam
Adapts the step size per direction from a running estimate of each gradient’s magnitude — big steps where shallow, small where steep.
optimizers.js — same gradient, three update rules
Ready · step 0
All three optimizers start at the same point on an ill-conditioned ravine — steep top-to-bottom, shallow left-to-right. Press Step to take one update each, or Run to animate. Watch SGD zigzag, Momentum accelerate, and Adam head straight for the bottom.
3.72
SGD loss
3.72
Momentum loss
3.72
Adam loss
How it works
The three optimizers all consume the same gradient at each step; they differ only in how they turn it into a move, and that difference is exactly a response to ill-conditioning (a large ratio between the steepest and shallowest directions of the surface). SGD uses the raw gradient: θ ← θ − η·g. Its problem in a ravine is that the learning rate η is bounded by the steep direction — set it any higher and the steep direction diverges — but that same small η makes progress along the shallow direction painfully slow, and because the gradient keeps pointing across the valley, the path zigzags. Momentum keeps a velocity v ← μ·v − η·g; θ ← θ + v: the across-the-valley components alternate sign and cancel in the running velocity, while the down-the-valley component is consistent and accumulates, so it effectively gets a much larger step in the direction that matters (at the cost of sometimes overshooting, which you can see as it rounds the bend). Adam tackles the conditioning directly by maintaining, per parameter, an exponential moving average of the gradient (a smoothed direction, like momentum) and of its squared magnitude, then dividing the step by the square root of that magnitude estimate: θ ← θ − η·m̂ / (√v̂ + ε). This normalizes each direction to a similar scale, so the steep and shallow directions get comparable-sized steps and the effective path heads much more directly toward the minimum — which is why Adam and its variants are the default for training large models, though well-tuned SGD-with-momentum still generalizes better on some vision tasks. The meta-lesson: the loss surface’s shape, not just its height, governs training, and optimizers are tools for coping with that shape.
1
All three read the same gradient
At the shared starting point, every optimizer computes the same gradient of the loss. In a ravine that gradient points mostly across the narrow, steep direction — not toward the distant minimum along the floor.
2
SGD steps raw → zigzag
SGD moves straight opposite the gradient. The learning rate is capped by the steep direction, so it overshoots across the valley and only inches along the floor, tracing a zigzag.
3
Momentum accumulates velocity
Momentum adds each step to a running velocity. The across-valley components alternate and cancel; the along-valley component builds up — so momentum accelerates down the floor and damps the bouncing (sometimes overshooting the bend).
✓
Adam adapts per direction
Adam divides each direction’s step by an estimate of its gradient magnitude, giving steep and shallow directions comparable steps. It heads almost straight for the minimum — which is why it’s the default optimizer for large models.
Surface
ill-conditioned ravine
SGD
raw gradient → zigzag
Momentum
velocity → accelerate
Adam
per-direction rescale
The code
# Same gradient g at each step — three ways to use it# SGD
theta -= lr * g
# Momentum (accumulate a velocity)
v = mu * v - lr * g
theta += v
# Adam (per-direction adaptive step)
m = b1*m + (1-b1)*g # smoothed gradient (like momentum)
s = b2*s + (1-b2)*g*g # smoothed squared magnitude
m_hat = m / (1 - b1**t); s_hat = s / (1 - b2**t) # bias-correct
theta -= lr * m_hat / (sqrt(s_hat) + eps) # rescale each direction
Quick check
1. Why does plain SGD zigzag on an ill-conditioned (ravine-shaped) loss surface?
In a ravine the gradient points mostly across the steep, narrow direction. The learning rate must stay small enough not to diverge there, but that same small rate makes progress along the shallow direction slow — so SGD bounces from wall to wall and inches toward the minimum.
2. How does momentum speed up descent along a ravine?
Momentum keeps a running velocity. The across-the-valley gradient components alternate in sign and cancel out in the velocity, while the down-the-valley component is consistent and builds up — giving a much larger effective step in the useful direction (sometimes overshooting the bend).
3. What is Adam’s key idea beyond momentum?
Adam maintains, per parameter, a smoothed gradient (like momentum) and a smoothed squared magnitude, then divides the step by the square root of that magnitude. This normalizes each direction to a similar scale, so ill-conditioning hurts far less and the path heads more directly to the minimum.
FAQ
What is the difference between SGD, momentum, and Adam?
Three rules for turning a gradient into an update. SGD steps directly opposite the gradient — simple but slow and zigzaggy on ill-conditioned surfaces. Momentum adds each step to a running velocity so consistent directions accelerate and oscillations cancel, speeding descent along ravines. Adam adapts the step per parameter using running estimates of each gradient’s mean and squared magnitude, normalizing directions. Adam is the common default for large models; well-tuned SGD+momentum sometimes generalizes better.
What does it mean for a loss surface to be ill-conditioned?
An ill-conditioned surface is far steeper in some directions than others (a high condition number), geometrically a long narrow ravine. It hurts gradient descent because one learning rate serves all directions: small enough not to diverge in the steepest direction means crawling in the shallow ones. Real neural-network loss surfaces are badly ill-conditioned, which is exactly why momentum and adaptive optimizers like Adam matter so much.
Why is Adam the default optimizer for training large models?
Adam works well with little tuning, which matters when a training run is expensive. Its per-parameter adaptive steps handle the very different gradient scales across a large model, making steady progress without a hand-scheduled learning rate, and it’s robust to sparse, noisy gradients. Variants like AdamW (fixing weight-decay interaction) are standard for transformers. The caveat: on some problems, especially vision, well-tuned SGD+momentum generalizes slightly better — so Adam is the pragmatic default, not a universal winner.
What is the learning rate and why does it matter so much?
The learning rate is the size of the step in the update direction — the single most important training hyperparameter. Too large and the loss oscillates or diverges (especially in steep directions); too small and training crawls or stalls. On an ill-conditioned surface a fixed rate can’t suit every direction at once, which is part of what momentum and Adam address. In practice people use schedules — warmup then decay — because the ideal step size changes as parameters near a minimum.