ReLU

ReLU (Rectified Linear Unit) is the activation function max(0, x): it passes positive inputs through unchanged and clamps negatives to zero. Its simplicity and cheap gradient made deep networks practical to train.

Worked example: for inputs [-2, 0.5, 3] ReLU outputs [0, 0.5, 3]; its gradient is 1 for positive inputs and 0 for negative, so it does not squash gradients the way sigmoid does. Gotcha: a unit that only ever receives negative inputs outputs zero forever with zero gradient — a “dead ReLU” — which is why leaky-ReLU (a small negative slope) and GELU exist.