Why does the derivative of the activation matter?

Backpropagation multiplies gradients layer by layer, and each layer's gradient includes the activation's derivative. If that derivative is tiny, the gradient shrinks as it flows backward — the vanishing-gradient problem — and early layers barely learn. Activations with a healthy derivative over a wide range let the signal reach deep layers.

Why did ReLU become the default?

ReLU (max(0, x)) has a derivative of exactly 1 for all positive inputs, so gradients pass through undiminished — no saturation on the positive side. It is also cheap to compute. That made much deeper networks trainable, which is a big part of why deep learning took off. Its one weakness, dead neurons for negative inputs, is patched by leaky ReLU and GELU.

What is GELU and where is it used?

GELU (Gaussian Error Linear Unit) is a smooth activation that behaves like ReLU for large inputs but curves gently near zero and allows small negative outputs. Its smoothness helps optimization, and it is the default activation inside most Transformers, including GPT and BERT.

LABS · 09 / ACTIVATIONS

The Bend.

Q: What is an activation function?

An activation function is a non-linear function applied to each neuron's output in a neural network. Without it, stacking layers would just compose linear operations into one linear operation, no matter how deep — so the network could only learn straight-line relationships. The non-linearity is what lets deep networks approximate complex, curved functions.

Q: What is the vanishing-gradient problem?

Saturating activations like sigmoid and tanh flatten out for large positive or negative inputs, so their derivative there is nearly zero. In a deep network, multiplying many near-zero derivatives during backprop drives the gradient toward zero, so early layers get almost no learning signal. This stalled training in deep networks before ReLU.

Stack a hundred layers of pure linear algebra and you still get… a straight line. The bend — a non-linear activation on every neuron — is what lets a network learn curves. Slide a point along each one and read its gradient: where it's alive, and where it dies.

THE GIST · 20 SECONDS

An activation function adds non-linearity to each neuron — without it, deep layers collapse into one linear map. Its derivative is what backprop multiplies to learn, so a flat region means a dying gradient. Sigmoid/tanh saturate (slope → 0 at the edges); ReLU keeps a slope of 1 for x > 0; GELU is a smooth modern default.

Non-linearlets nets learn curves
Derivativewhat backprop uses
Saturateslope → 0 = no learning
ReLUslope 1, no vanish

Pick a function, then drag the x slider to move the point — the tangent's steepness is the gradient backprop would use there.

UNDER THE HOOD

What you just played, written down

The shape of an activation decides what a network can represent; the shape of its derivative decides whether the network can actually learn it.

Why activations matter — four ideas

Linearity collapses. Stacking linear layers without a bend equals one linear layer — no matter the depth.
The bend adds power. A non-linear activation lets layers combine into arbitrarily complex, curved functions.
Backprop multiplies derivatives. Each layer's gradient carries the activation's slope, chained back through the network.
Flat slope = no learning. Where the derivative is near zero (saturation), the gradient vanishes and those weights barely move.

THE ReLU REVOLUTION

Sigmoid's derivative peaks at just 0.25 and falls to ~0 at the edges — stack a few layers and the gradient dwindles. ReLU's derivative is a flat 1 for x > 0, so gradients pass through undimmed. That's a big reason deep nets became trainable.

The functions in code

import numpy as np

def sigmoid(x): return 1 / (1 + np.exp(-x))   # (0, 1)
def tanh(x):    return np.tanh(x)             # (-1, 1)
def relu(x):    return np.maximum(0, x)       # slope 1 for x>0
def leaky(x):   return np.where(x > 0, x, 0.1 * x)
def gelu(x):
    return 0.5 * x * (1 + np.tanh(
        np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))

# derivatives are what backprop multiplies:
d_sigmoid = lambda x: (s := sigmoid(x)) * (1 - s)  # max 0.25
d_relu    = lambda x: (x > 0).astype(float)        # 1 or 0

SIGMOID f'ₘₐₓ0.25vanishes at edges

ReLU f' (x>0)1.0no saturation

DEAD ReLUs → LEAKY & GELU

ReLU's flat zero for x < 0 can leave neurons dead (always outputting 0). Leaky ReLU gives a small negative slope; GELU curves smoothly and is the default inside Transformers.

⚠ Softmax is different

Softmax isn't a per-neuron activation — it's applied across a whole output layer to produce a probability distribution. It's a normalizer for classification, not a hidden-layer bend.

↔ Picking one

ReLU is a safe default for hidden layers; GELU for Transformers; sigmoid for a single probability output; tanh when you want zero-centered outputs. Rarely sigmoid/tanh in deep hidden layers.

★ Why it's everywhere

Every neuron in every neural network passes through one of these. The choice affects how fast — and how deep — a model can learn. It's a small function with an outsized impact.

QUICK CHECK

Did it stick?

FAQ

Activations, answered

What is an activation function?

A non-linear function applied to each neuron's output. Without it, stacked layers collapse into a single linear map, so the network could only learn straight lines. The bend is what enables deep networks to model curves.

Why does its derivative matter?

Backprop multiplies gradients layer by layer, including the activation's derivative. A near-zero derivative shrinks the gradient as it flows back — early layers stop learning.

What is the vanishing-gradient problem?

Saturating activations (sigmoid, tanh) flatten at the extremes, so their derivative is ~0 there. Multiplying many tiny derivatives through a deep net drives the gradient to zero, stalling training.

Why did ReLU win?

Its derivative is a flat 1 for positive inputs, so gradients pass undiminished, and it's cheap to compute — making much deeper networks trainable. Leaky ReLU and GELU patch its dead-neuron weakness.

What is GELU?

A smooth activation that acts like ReLU for large inputs but curves gently near zero and allows small negatives. It's the default inside most Transformers, including GPT and BERT.

Finished this one? 0 / 12 Labs done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

ML Foundations

The Bend.

What you just played, written down

Why activations matter — four ideas

The functions in code

⚠ Softmax is different

↔ Picking one

★ Why it's everywhere

Did it stick?

Activations, answered

Explore the topic

More Labs

Explore more from Vibe Engines

What you just played, written down

Why activations matter — four ideas

The functions in code

⚠ Softmax is different

↔ Picking one

★ Why it's everywhere

Did it stick?

Activations, answered

Explore the topic

More Labs

Explore more from Vibe Engines

Get the next one in your inbox.