Activation Function
also: ReLU · GELU
The nonlinear function at each neuron that lets a network learn more than straight lines.
An activation function is the nonlinearity applied to a neuron’s weighted sum. Without it, stacking layers would collapse into a single linear map; the nonlinearity is what lets deep networks approximate complex functions. Common ones: ReLU, GELU, sigmoid, tanh.
Worked example: ReLU (max(0, x)) is the workhorse — cheap, and it avoids the vanishing-gradient problem of sigmoid/tanh for positive inputs; transformers typically use GELU, a smoother variant. Gotcha: ReLU units can “die” (stuck at zero if they only ever see negative inputs), which is why variants like leaky-ReLU and GELU exist.