LABS · 22  /  NEURAL NETS

The Blame Machine.

A network makes a guess and gets it wrong. Now the hard question: which weight is to blame, and by how much? Backpropagation answers it for every weight at once — pushing the error backward through the network with the chain rule. Run the forward pass, feel the loss, then watch the blame flow back.

Act 1 · One pass each way
Guess, measure, assign blame
A tiny network turns an input into a prediction (forward, cyan). We compare it to the target — that's the loss. Then the chain rule carries the error backward (orange), handing each weight a gradient: how much nudging it would change the loss. Step through it.
Forward (activations) Backward (gradients) Loss
Prediction
target = 1.0
Loss
idle
Phase
Forward is easy — just multiply and add through the layers. The magic is backward: one sweep hands every weight its share of the blame, in about the cost of the forward pass. That efficiency is why deep learning is possible at all.
Act 2 · Gradient descent
Nudge every weight downhill
Blame in hand, each weight steps opposite its gradient — the direction that lowers the loss. Do it over and over and the network learns. Set the learning rate and hit train: watch the loss fall, or, if you crank it, watch it explode.
Learning rate0.30
Current loss
0
Steps
ready
Status
This loop — forward, backprop, nudge — repeated billions of times, is literally how every neural network learns, from a toy like this to a frontier LLM. The learning rate is the step size: too small crawls, too big diverges.
Act 3 · The depth problem
Why deep networks were hard
Backprop multiplies a local derivative at every layer. In a deep net that's many factors multiplied together — and multiplication is unstable. If each factor is under 1, the gradient vanishes to nothing; over 1, it explodes. Slide the depth and the per-layer factor to watch early layers go dark or blow up.
Network depth12
Per-layer gradient factor0.8
Gradient at layer 1
Diagnosis
The fix
This is why deep nets were nearly untrainable for years. The fixes — ReLU activations, normalization layers, residual connections, gradient clipping — all keep that per-layer factor near 1 so the product stays sane across dozens or hundreds of layers.

The whole of deep learning is one derivative

Training a neural network is an optimization problem: find the weights that make the loss small. To descend the loss you need its gradient — how it changes as you nudge each weight. The naive way (perturb one weight, re-run the network, measure the change, repeat for millions of weights) is hopeless. Backpropagation computes all the gradients in one backward sweep, for about the cost of one forward pass. That efficiency is the whole ballgame.

Forward, then backward

The forward pass runs the input through the layers to a prediction, then a loss measures the error. The backward pass applies the chain rule: a network is a chain of functions, so the loss's sensitivity to an early weight is the product of the local derivatives along the path from that weight to the loss. Backprop computes each local derivative once and multiplies them on the way back, reusing shared sub-results — that reuse is what makes it fast.

Gradient = blame

A weight's gradient is literally "how much did you contribute to the error, and in which direction?" Gradient descent then nudges each weight opposite its gradient — the direction that lowers the loss — scaled by the learning rate. Forward, backprop, nudge, repeat: that loop is how every model from a toy net to a frontier LLM is trained.

Where it breaks: the product of many factors

Backprop's engine — multiplying local derivatives along a path — is also its weakness. In a deep network that path is long, so the gradient at an early layer is a product of many numbers. If they're consistently below 1, the product vanishes toward zero and early layers barely learn; above 1, it explodes and training destabilizes. Both get worse with depth.

Nearly every architectural advance that unlocked deep networks is, underneath, a way to keep that per-layer factor near 1: ReLU and friends (derivative of 1 in the active region, dodging the saturating tails of sigmoid/tanh), normalization layers (keeping activations well-scaled), residual connections (a gradient shortcut straight back, the core trick of ResNet), and gradient clipping (capping explosions). The descent is what learns; backprop is what makes the descent computable.

Check yourself

Four questions. Propagate carefully.

Finished this one? 0 / 22 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

More Labs