GELU
also: gaussian error linear unit
A smooth version of ReLU used inside most transformers.
GELU (Gaussian Error Linear Unit) is a smooth activation that weights each input by roughly the probability it is positive, instead of ReLU’s hard cutoff at zero. It is the default activation inside most transformer feed-forward layers.
Worked example: near zero GELU curves gently — small negative inputs get a small negative output rather than a hard zero — giving a smoother gradient than ReLU’s kink. Gotcha: the difference from ReLU is small in isolation but consistently helps at transformer scale; it is one of several “smooth ReLU” variants (SiLU/Swish is another) chosen empirically, not from deep theory.