When people say a model is “learning,” what’s literally happening is a point sliding downhill on a loss landscape — a surface whose height is the model’s error and whose location is its parameters. Gradient descent only ever does one thing: read the slope where it stands and step downhill. That means the shape of the landscape, not the algorithm, decides what answer you get. On a convex bowl there’s a single valley, so descent always rolls to the one true bottom — this is the easy, well-behaved case. On a non-convex surface with several valleys, descent rolls into whichever valley it happens to be above, so two runs starting from different points can end at different answers, and you can get trapped in a local minimum that’s worse than the best one elsewhere. And on a flat plateau the slope nearly vanishes, so each step is tiny and progress crawls to a near-halt even though you’re not at a minimum — the practical face of the “saddle points” that dominate high-dimensional loss surfaces. Drop a ball on each surface, change where it starts, and watch the same descent rule produce completely different fates.
gradient descent reads the slope · convex bowl always solves · non-convex can trap · plateaus starve the gradient
loss landscape
The surface of error over the model’s parameters. Training slides downhill on it toward low error.
convex
A single-valley shape: every downhill path reaches the one global minimum. The easy case.
local minimum
A valley that isn’t the lowest. Descent can settle here and get stuck, missing a better minimum elsewhere.
plateau / saddle
A near-flat region where the gradient nearly vanishes, so steps become tiny and progress stalls.
descent.js — same rule, different terrain
Ready
Pick a surface, then drop a ball and let gradient descent roll it downhill. On the convex bowl it always finds the bottom. On the double-well, where you drop it decides which minimum it reaches. On the plateau, watch the gradient — and the steps — nearly vanish.
—
position
—
loss
—
|gradient|
How it works
Gradient descent is a local procedure — it only ever knows the slope at its current point — which is the root of both its power and its limits. Because it uses local information, it scales to billions of parameters (you never need a global view of the surface), but for the same reason it is at the mercy of the landscape’s shape. On a convex surface, local downhill is globally correct: there is one minimum and every descent path reaches it, which is why classical convex problems (linear/logistic regression, SVMs) are considered “solved.” Neural-network losses are decidedly non-convex, full of many valleys — so descent finds a minimum determined by where it started and the path it took, not the minimum. For a long time people feared this meant training would get stuck in bad local minima, but a key insight from high-dimensional geometry changed the picture: in very high dimensions, points where the gradient is zero are overwhelmingly saddle points (down in some directions, up in others) rather than true local minima, and the local minima that do exist tend to have similar, near-optimal loss. The practical villain, then, isn’t a bad basin you can’t escape — it’s the flat regions and plateaus around saddles where the gradient nearly vanishes and progress grinds to a crawl. This reframes what optimizers are for: momentum, adaptive methods like Adam, good initialization, and normalization are largely about moving efficiently across flat regions and ill-conditioned valleys, not about heroically escaping deep local minima. And it’s why the starting point (initialization) genuinely matters, and why a little stochastic noise in the gradient can help — it nudges the ball off plateaus and out of shallow traps.
1
Descent reads the local slope
At its current point the ball measures the gradient (the slope) and steps downhill, proportional to the learning rate. It never sees the whole surface — only where it stands.
2
Convex bowl → always the bottom
With a single valley, every downhill path leads to the one global minimum. Drop the ball anywhere and it converges to the same, best answer. This is the easy, solvable case.
3
Two wells → the start decides
With several valleys, the ball rolls into whichever one it’s above. Drop it on the right and it can settle in a local minimum that’s worse than the deeper valley on the left — a trap that depends entirely on initialization.
✓
Plateau → the gradient vanishes
On a flat region the slope is nearly zero, so each step is tiny and progress stalls even though you’re not at a minimum. This is the practical face of the saddle points that dominate high-dimensional loss surfaces — and what optimizers mainly fight.
Descent uses
only the local slope
Convex
one minimum, always found
Non-convex
start decides the minimum
Plateau/saddle
gradient ≈ 0 → stalls
The code
# Gradient descent: the SAME rule on any surface
x = start
for step in range(N):
g = gradient(loss, x) # only the LOCAL slope is known
x = x - lr * g # step downhillif abs(g) < tiny: # plateau / near-saddle:break# steps shrink to nothing — stalled# convex loss -> any start reaches the one global minimum# non-convex -> the start point decides WHICH minimum you get
Quick check
1. On a convex (single-valley) loss surface, where does gradient descent end up?
A convex surface has exactly one valley, so local downhill is globally correct: no matter where you start, descent reaches the single global minimum. This is why convex problems like linear and logistic regression are considered reliably solvable.
2. Why can two training runs on a non-convex surface reach different answers?
Gradient descent is local — it only follows the slope where it stands. On a surface with several valleys, it settles into the one it happens to be above, so different starting points can lead to different minima, some worse than others. That’s why initialization matters.
3. In practice, what most slows training on high-dimensional loss surfaces?
In high dimensions, zero-gradient points are overwhelmingly saddle points (down in some directions, up in others) rather than bad local minima, and most true minima have similar near-optimal loss. The real obstacle is the flat plateaus around saddles, where the gradient nearly vanishes and descent crawls — which is largely what momentum and adaptive optimizers exist to overcome.
FAQ
What is a loss landscape?
The surface you get by plotting a model’s loss (error) against its parameters: each parameter setting is a point and its height is the error there. Training moves across this surface toward lower points via gradient descent. Its shape — a smooth bowl, a rugged surface of many valleys, or flat plateaus — determines how hard training is and what solution you reach. For real networks it lives in millions of dimensions, so we only visualize simplified slices.
Do neural networks get stuck in bad local minima?
Much less than once feared. The old worry was that non-convex training would trap descent in poor minima, but high-dimensional geometry showed that zero-gradient points are overwhelmingly saddle points, not minima, and the minima that exist tend to have similar near-optimal loss. So the practical difficulty is moving through flat regions and saddles where the gradient nearly vanishes, not escaping deep bad basins. Initialization and optimizer choice still matter, and bad basins can occur in small or pathological models.
What is a saddle point and why does it slow training?
A saddle point is where the gradient is zero but it isn’t a minimum — the surface descends in some directions and rises in others, like a horse’s saddle. It slows training because the gradient is tiny in many directions nearby, so descent takes minuscule steps and creeps across the flat region before finding a clearly downhill direction. In high dimensions, saddles and their plateaus are far more common than true local minima, making them the main obstacle to fast training — which is why momentum and adaptive optimizers help.
Why does the starting point (initialization) matter in training?
Because gradient descent is local — it follows the slope from wherever it starts, so the initial parameters shape which region it explores and, on a non-convex surface, which minimum it can reach. Initialization also drives early dynamics: a poor one can land you on a vanishing-gradient plateau or where activations explode or die, stalling learning. That’s why principled schemes (Xavier/Glorot, He) set the initial scale so gradients behave, and why normalization and learning-rate warmup keep early training stable.