A network makes a guess and gets it wrong. Now the hard question: which weight is to blame, and by how much?Backpropagation answers it for every weight at once — pushing the error backward through the network with the chain rule. Run the forward pass, feel the loss, then watch the blame flow back.
Act 1 · One pass each way
Guess, measure, assign blame
A tiny network turns an input into a prediction (forward, cyan). We compare it to the target — that's the loss. Then the chain rule carries the error backward (orange), handing each weight a gradient: how much nudging it would change the loss. Step through it.
Forward (activations) Backward (gradients) Loss
—
Prediction
target = 1.0
—
Loss
idle
Phase
Forward is easy — just multiply and add through the layers. The magic is backward: one sweep hands every weight its share of the blame, in about the cost of the forward pass. That efficiency is why deep learning is possible at all.
Blame in hand, each weight steps opposite its gradient — the direction that lowers the loss. Do it over and over and the network learns. Set the learning rate and hit train: watch the loss fall, or, if you crank it, watch it explode.
Learning rate0.30
—
Current loss
0
Steps
ready
Status
This loop — forward, backprop, nudge — repeated billions of times, is literally how every neural network learns, from a toy like this to a frontier LLM. The learning rate is the step size: too small crawls, too big diverges.
Act 3 · The depth problem
Why deep networks were hard
Backprop multiplies a local derivative at every layer. In a deep net that's many factors multiplied together — and multiplication is unstable. If each factor is under 1, the gradient vanishes to nothing; over 1, it explodes. Slide the depth and the per-layer factor to watch early layers go dark or blow up.
Network depth12
Per-layer gradient factor0.8
—
Gradient at layer 1
—
Diagnosis
—
The fix
This is why deep nets were nearly untrainable for years. The fixes — ReLU activations, normalization layers, residual connections, gradient clipping — all keep that per-layer factor near 1 so the product stays sane across dozens or hundreds of layers.
The whole of deep learning is one derivative
Training a neural network is an optimization problem: find the weights that make the loss small. To descend the loss you need its gradient — how it changes as you nudge each weight. The naive way (perturb one weight, re-run the network, measure the change, repeat for millions of weights) is hopeless. Backpropagation computes all the gradients in one backward sweep, for a small multiple of the cost of one forward pass (the backward sweep does roughly twice the forward's arithmetic). That efficiency is the whole ballgame.
Forward, then backward
The forward pass runs the input through the layers to a prediction, then a loss measures the error. The backward pass applies the chain rule: a network is a chain of functions, so the loss's sensitivity to an early weight is the product of the local derivatives along the path from that weight to the loss. Backprop computes each local derivative once and multiplies them on the way back, reusing shared sub-results — that reuse is what makes it fast.
Gradient = blame
A weight's gradient is literally "how much did you contribute to the error, and in which direction?" Gradient descent then nudges each weight opposite its gradient — the direction that lowers the loss — scaled by the learning rate. Forward, backprop, nudge, repeat: that loop is how every model from a toy net to a frontier LLM is trained.
Where it breaks: the product of many factors
Backprop's engine — multiplying local derivatives along a path — is also its weakness. In a deep network that path is long, so the gradient at an early layer is a product of many numbers. If they're consistently below 1, the product vanishes toward zero and early layers barely learn; above 1, it explodes and training destabilizes. Both get worse with depth.
Nearly every architectural advance that unlocked deep networks is, underneath, a way to keep that per-layer factor near 1: ReLU and friends (derivative of 1 in the active region, dodging the saturating tails of sigmoid/tanh), normalization layers (keeping activations well-scaled), residual connections (a gradient shortcut straight back, the core trick of ResNet), and gradient clipping (capping explosions). The descent is what learns; backprop is what makes the descent computable.
Backpropagation, step by step
Reverse-mode differentiation is four mechanical steps. Below they are spelled out, then run by hand on the exact chain in Act 1, so the prediction, loss and gradients the widget draws are written here too.
The algorithm
Forward, and keep everything. Run the input through every layer and cache each intermediate value. The backward pass needs them.
Seed the gradient at the loss. For squared error L = (ŷ − y)², the first gradient is dL/dŷ = 2(ŷ − y).
Walk back one node at a time. Multiply the incoming gradient by each local derivative: dL/dw = dL/dout · dout/dw for the weight, and dL/din = dL/dout · dout/din to pass further back. For a matrix layer y = Wx that is dL/dW = dL/dy · xᵀ and dL/dx = Wᵀ · dL/dy.
Update. Step every weight against its gradient: w = w − η · dL/dw.
Worked example: the widget's chain
Forward. x = 1.0, w₁ = 0.9, w₂ = 0.8, w₃ = 1.1. Then h₁ = 0.9, h₂ = 0.8 × 0.9 = 0.72, ŷ = 1.1 × 0.72 = 0.792. Target 1.0, so the loss is (0.792 − 1)² = 0.043.
w₂ and w₁. dL/dw₂ = −0.458 × h₁ = −0.412, and dL/dh₁ = −0.458 × w₂ = −0.366, so dL/dw₁ = −0.366 × x = −0.366. All three are negative: raising any weight lowers the loss.
Check it numerically. Nudging w₁ by ±0.001 and dividing the change in loss by 0.002 gives −0.36608, matching backprop to 14 significant digits.
One step at η = 0.3, the widget's default: the weights become 1.010, 0.924 and 1.190, the prediction overshoots to 1.110, and the loss drops to 0.012. Five steps take it to 0.0027 as the prediction swings around 1.
Knobs and pitfalls
Cost. The backward pass does about twice the forward's arithmetic (a matrix layer needs one product for dL/dW and another for dL/dx), so a training step costs about three forward passes. Memory is the real limit: cached activations grow with batch size, depth and sequence length. Activation checkpointing stores only some of them and recomputes the rest, about one extra forward pass for a large memory saving.
Stability. Clip the global gradient norm (1.0 is a common setting for large language models) to cap explosions, and in fp16 training scale the loss up so small gradients do not underflow to zero. After writing a custom layer, always compare its gradients with finite differences, as above, in float64.
Check yourself
Four questions. Propagate carefully.
See every answer, with the reasoning
What does backpropagation compute?
Answer: A gradient for every weight — how much each contributed to the loss — It assigns blame to every weight in one backward sweep, via the chain rule.
Why is backprop efficient?
Answer: It computes local derivatives once and reuses shared sub-results down the chain — one backward pass — All gradients for about the cost of one forward pass — versus perturbing each weight separately.
What causes vanishing/exploding gradients?
Answer: Multiplying many per-layer derivatives along a deep path — under 1 shrinks to zero, over 1 blows up — Backprop’s product-of-factors is unstable with depth; both failure modes worsen as networks get deeper.
How do ReLU, normalization and residual connections help?
Answer: They keep the per-layer gradient factor near 1 so the product stays sane across depth — All are ways to keep gradients healthy through many layers — the reason deep nets became trainable.