Feed-Forward Network
also: FFN · MLP block
The per-token MLP inside each transformer block — where much of the model’s knowledge is stored.
A feed-forward network (FFN) is the position-wise multilayer perceptron inside each transformer block: after attention mixes information across tokens, the FFN processes each token independently through an expand-then-contract MLP (a large hidden layer with a nonlinearity). It is where a large share of a model’s parameters — and its stored knowledge — live.
Worked example: in a transformer block, attention decides which other tokens each token should read, then the FFN transforms each token’s representation — typically expanding to about 4× the model dimension, applying GELU, and projecting back. Gotcha: the FFN holds most of the parameters, which is exactly what mixture-of-experts targets — replacing the single dense FFN with many expert FFNs and routing each token to a few — to grow capacity without growing per-token compute.