What is a local minimum in gradient descent?

A local minimum is a valley in the loss landscape that is lower than its immediate surroundings but not the lowest point overall (the global minimum). Plain gradient descent can get stuck in a local minimum because the slope there is zero, so it stops moving. Momentum, random restarts, and stochastic noise help the optimizer roll through shallow local minima toward better solutions.

What is the difference between gradient descent and stochastic gradient descent (SGD)?

Batch gradient descent computes the gradient over the entire dataset before each step — accurate but slow and memory-heavy. Stochastic gradient descent (SGD) estimates the gradient from one example (or a small mini-batch) per step, so it takes many more, noisier steps that are far cheaper and often escape shallow local minima. Mini-batch SGD is what actually trains modern neural networks.

What is momentum in gradient descent?

Momentum adds a fraction of the previous update to the current one, so the optimizer accumulates velocity in consistent directions — like a ball rolling downhill. It speeds up convergence along gentle slopes, damps oscillation across steep valleys, and carries the optimizer through shallow local minima. Optimizers like Adam combine momentum with a per-parameter adaptive learning rate.

Vibe Engines is free, interactive AI engineering & system design — runs in your browser. Pick a learning path

LABS · 01 / GRADIENT DESCENT

The Descent.

Q: What is gradient descent?

Gradient descent is the optimization algorithm used to train most machine-learning models, including neural networks. It minimizes a loss function by repeatedly moving the parameters a small step in the direction of steepest descent — the negative gradient (slope). Over many steps it rolls 'downhill' on the loss landscape toward a minimum, which corresponds to the model's best settings.

Q: What does the learning rate do?

The learning rate (often written η or alpha) is the step size of gradient descent: how far the parameters move along the slope each iteration. Too small and training crawls and may never finish; too large and the steps overshoot the minimum, oscillate, or diverge entirely — the loss blows up to infinity. Choosing or scheduling the learning rate is the single most important hyperparameter in training.

Q: Why does training diverge or the loss become NaN?

The most common cause is a learning rate that is too high: each step overshoots so far that the loss increases instead of decreasing, compounding until it overflows to infinity or NaN. Fixes include lowering the learning rate, using a warmup or decay schedule, clipping gradients, and normalizing inputs. Exploding gradients in deep networks have the same effect and are addressed with gradient clipping and normalization layers.

Every neural network and language model is trained by rolling downhill on a loss landscape. You'll do it by hand and waste steps — then meet the algorithm that just reads the slope, and watch a bad learning rate blow the whole thing up.

THE GIST · 20 SECONDS

Gradient descent trains a model by minimizing a loss — a number that says how wrong it is. It treats the loss as a landscape over the model's parameters and repeatedly steps in the direction of steepest descent (the negative gradient, i.e. the downhill slope). The step size is the learning rate. Roll downhill enough times and you reach a minimum — the model's best settings.

Losshow wrong the model is — lower is better
Gradientthe slope; points uphill, so we go negative
Learning ratestep size each iteration (η)
Minimumthe bottom of a valley — the goal

① Descend by hand → ② Follow the slope → ③ Tune the learning rate till it explodes → ④ Escape a local minimum

parameter (the ball) slope / step minimum

The height of the curve is the loss; the ball's left–right position is the model's parameter. Training = rolling the ball to the lowest point by following the downhill slope.

UNDER THE HOOD

What you just played, written down

You rolled it by hand, followed the slope, tuned the step size, and escaped a trap. Here's the same thing as the rule that trains every neural network.

How the optimizer thinks — four steps

Measure the loss. Run the model, compare to the target, get a single number — how wrong it is.
Take the gradient. Compute the slope of the loss with respect to each parameter — the direction that increases loss fastest.
Step downhill. Move each parameter a little in the opposite direction: w ← w − η·∇L. η is the learning rate.
Repeat until the gradient is ~0 (a minimum) or you run out of steps.

WHY THE LEARNING RATE IS EVERYTHING

Each step moves by η × slope. Too small → it crawls and may never arrive. Too big → it leaps past the bottom and the loss grows instead of shrinks, spiralling to infinity. The whole art of training lives in this one number.

The update rule

for each step:
    L      = loss(params)          # how wrong we are
    grad   = ∇L(params)            # slope of L
    params = params - lr * grad    # step downhill

# with momentum (a rolling ball):
    v      = beta * v - lr * grad
    params = params + v

CONVERGES WHENη small enoughloss decreases every step

DIVERGES WHENη too largeovershoots · loss → ∞

Solved by hand — gradient descent on the valley above

Each step, the parameter, its loss, and the slope the optimizer read — with a sensible learning rate. Watch the slope shrink toward zero as it nears the bottom.

⚠ When it breaks

A learning rate that's too high makes the loss explode to NaN; a local minimum can trap plain descent because the slope there is zero. Momentum and good initialization help.

↔ Its cousins

SGD estimates the slope from a mini-batch (cheaper, noisier, escapes shallow traps). Momentum adds velocity; Adam adds a per-parameter adaptive learning rate.

★ Where you've used it

Every trained model: neural networks and LLMs, logistic/linear regression, recommendation systems, and fine-tuning — all minimize a loss with some flavour of gradient descent.

STEP BY STEP

Gradient descent, step by step

Why one learning rate crawls, another lands in a single step and a third explodes, worked out on the exact valley in the widget, L(x) = 1.3(x − 0.66)² + 0.05, starting from x = 0.12.

The algorithm, and its one condition

Start from a guess θ₀ and pick a learning rate η.
Read the slope. Compute g = ∇L(θ); for a network that is backpropagation. On this valley, g = 2.6(x − 0.66).
Step. θ ← θ − η·g. On a quadratic with curvature c (here 2.6), every step multiplies the distance to the minimum by (1 − η·c).
Read that factor. Between 0 and 1 it crawls in from one side; between −1 and 0 it overshoots but still converges; below −1 it diverges. So η must stay under 2/c = 0.769 here, and η = 1/c = 0.385 lands in one step.

# error factor on a quadratic
e_next = (1 - lr * c) * e
# Adam, at step t
m = b1*m + (1 - b1)*grad
s = b2*s + (1 - b2)*grad**2
m_hat = m / (1 - b1**t)
s_hat = s / (1 - b2**t)
theta -= lr*m_hat/(sqrt(s_hat)+eps)

Worked example: five learning rates

η = 0.18, the trace above: x goes 0.120 → 0.373 → 0.507 → 0.579 → 0.617 → 0.637, loss 0.429 → 0.157 → 0.080 → 0.059 → 0.052 → 0.051, slope −1.40 → −0.75 → −0.40 → −0.21 → −0.11 → −0.06. Each slope is 0.532 times the last; Act 2 stops after 9 steps at x = 0.66.
η = 0.05, factor 0.87: after six steps x is only 0.426, and it takes 46 steps to get within 0.001 of the minimum.
η = 0.385, factor 0: x = 0.66 after one step.
η = 0.6, factor −0.56: x goes 0.12 → 0.962 → 0.491 → 0.755 → 0.607, crossing the minimum every step, while the loss still falls, 0.429 → 0.169 → 0.087 → 0.062.
η = 0.9, factor −1.34: x goes 0.12 → 1.384 → −0.310 → 1.959 and the loss climbs 0.429 → 0.731 → 1.272 → 2.245. Every η above 0.769 diverges like this.

⚠ The steepest direction sets the limit

With many parameters, the most curved direction caps η below 2/c_max while the flattest one sets how slowly everything else converges. That ratio is why long, narrow valleys zig-zag; normalized inputs, momentum and Adam shrink it.

↔ Schedules and batches

Large models ramp η up over the first few thousand steps (warmup), then decay it, often on a cosine curve. A mini-batch gradient's noise falls as 1/√B, so 4× the batch halves it, and the linear scaling rule raises η in proportion to the batch size, up to a point.

★ In production

AdamW trains transformers: PyTorch's defaults are β₁ = 0.9 and β₂ = 0.999, and many LLM recipes use β₂ = 0.95. SGD with momentum 0.9 remains common for vision models. In high dimensions, saddle points slow training more often than bad local minima trap it.

THE PICTURE

The picture: one factor decides crawl, land or explode

Both halves show the same rule. Every step multiplies the distance to the minimum by f = 1 − η·c. With η = 0.18, f = 0.532, so each step covers about half the remaining gap. With η = 0.9, f = −1.34, so each step lands on the far side and further out than the one before. The right half places all five of the page’s learning rates on that one number line.

TRADE-OFFS

Cost and trade-offs: what each optimizer step costs

Here Time is the work per step, Space is the memory on top of the model’s P parameters. N is the dataset size and B the batch size.

Case	Time	Space	Why
Full-batch gradient descent	O(N·P): every example, every step	one gradient (P numbers)	The exact slope, but one step costs a whole pass over the data.
Mini-batch SGD	O(B·P) per step	one gradient, plus activations for B examples	Far cheaper steps. The noise falls as 1/√B, as the page says.
SGD + momentum	same as SGD, plus one vector add	+1 velocity vector (P numbers)	On a long, narrow valley, well-tuned momentum needs on the order of √κ steps instead of κ, where κ is the ratio of steepest to flattest curvature.
Adam / AdamW	same as SGD, plus a few element-wise ops	+2 vectors, m and s (2P numbers)	A step size per parameter. For a 7-billion-parameter model in 32-bit floats, m and s alone take 56 GB.
Newton’s method	O(P³) to solve with the Hessian	O(P²) for the Hessian	Lands on a quadratic’s minimum in one step. In 1-D it is gradient descent with η = 1/c, the 0.385 above. Impossible at P in the billions.

Steps matter as much as cost per step. On the page’s valley, reaching within 0.001 of the minimum takes 46 steps at η = 0.05, 10 at 0.18, 1 at 0.385, 11 at 0.6, and never at 0.9. Overshooting at 0.6 is almost as fast as creeping in at 0.18. Now add a second, flatter direction with curvature 0.026 (κ = 100). η must stay under 2/2.6 for the steep one, so even at η = 1.9/2.6 the flat direction’s factor is 0.981, and shrinking its error 1,000× takes 361 steps. That is the narrow-valley zig-zag the page warns about, and the reason momentum, Adam and normalised inputs exist.

RUN IT YOURSELF

Run gradient descent yourself

The lab above, in code: gradient descent minimising f(x) = (x − 3)² — real CPython in your browser. Change the learning rate and watch it crawl, converge, or diverge. Hit Run (or ⌘/Ctrl + Enter).

HOW TO READ THE CODE — 4 IDEAS

The gradient (slope) of f at x is f'(x) = 2(x − 3) — it points uphill.
Step the opposite way to go downhill: x ← x − lr · gradient (the update).
lr (learning rate) is the step size: too small crawls, too big overshoots or diverges.
Repeat and x homes in on the minimum, x = 3.

CPython · WebAssembly

QUICK CHECK

Did it stick?

See every answer, with the reasoning

In which direction does gradient descent move the parameters?
Answer: Opposite the gradient (downhill) — The gradient points uphill (steepest increase), so to minimize loss we step in the negative-gradient direction — downhill.
What happens if the learning rate is far too large?
Answer: The steps overshoot and the loss diverges to infinity — Too-large steps leap past the minimum, so the loss grows each step and compounds to infinity or NaN — the most common reason training blows up.
Why can plain gradient descent get stuck in a local minimum?
Answer: The slope (gradient) there is zero, so it stops moving — At any minimum the slope is zero, so the update is zero and plain descent halts — even if a deeper global minimum exists elsewhere. Momentum and noise help escape.

FAQ

Gradient descent, answered

What is gradient descent?

Gradient descent trains most ML models by minimizing a loss: it repeatedly steps the parameters in the direction of steepest descent (the negative gradient), rolling downhill on the loss landscape toward a minimum.

What does the learning rate do?

The learning rate (η) is the step size. Too small → training crawls; too large → steps overshoot and the loss diverges to infinity. It's the most important training hyperparameter.

What is a local minimum?

A valley that's lower than its surroundings but not the lowest overall (the global minimum). Plain descent can get stuck there because the slope is zero; momentum and noise help escape shallow ones.

Gradient descent vs stochastic gradient descent (SGD)?

Batch GD uses the whole dataset per step (accurate, slow). SGD uses one example or mini-batch per step (noisy, cheap, escapes shallow traps). Mini-batch SGD trains modern neural networks.

What is momentum?

Momentum adds a fraction of the last update to the next, accumulating velocity like a rolling ball — speeding convergence, damping oscillation, and carrying the optimizer through shallow local minima. Adam combines it with adaptive learning rates.

Why does training diverge or the loss become NaN?

Usually the learning rate is too high: each step overshoots, the loss grows, and it compounds to infinity/NaN. Lower the rate, add warmup/decay, clip gradients, and normalize inputs.

Finished this one? 0 / 61 Labs done

NextThe Blame Machine: Backpropagation Next on the “Training a network” ladder · rung 5 of 7

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

More Labs

Explore more

Algorithms →