CODING CHALLENGE · N°09

Layer Normalization

Easy AI EngineeringTransformersDeep Learning

The stabilizer wrapped around every Transformer sub-layer: re-center and re-scale a vector to mean 0 and variance 1, then let two learned parameters stretch and shift it back. Keeps activations well-behaved so deep nets actually train. No numpy — just the math.

The problem

Implement layer normalization over a single vector x. Compute the mean and (population) variance of x, normalize each element to zero mean and unit variance, then apply a per-element scale gamma and shift beta: y[i] = gamma[i] · (x[i] − mean) / √(var + eps) + beta[i]. Use eps = 1e-5 for numerical stability (so a constant input does not divide by zero). Return the list y.

EXAMPLE 1

Input x=[1,2,3,4], gamma=[1,1,1,1], beta=[0,0,0,0]

Output ≈[-1.34, -0.45, 0.45, 1.34]

mean 0, variance 1 — the raw normalize

EXAMPLE 2

Input x=[0,10], gamma=[1,1], beta=[0,0]

Output ≈[-1.0, 1.0]

two elements → symmetric ±1

EXAMPLE 3

Input x=[7,7,7], gamma=[1,1,1], beta=[0,0,0]

Output [0.0, 0.0, 0.0]

constant input → eps keeps it finite

CONSTRAINTS

Use the population variance: mean((x − mean)²), dividing by n (not n−1).
Add eps = 1e-5 inside the square root before dividing.
gamma and beta are per-element (same length as x) — scale then shift each normalized value.
Pyodide has no numpy — use the math stdlib and plain lists only.

SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement layer_norm(x, gamma, beta, eps=1e-5). Center and scale x to mean 0 / variance 1, then apply the per-element scale gamma and shift beta.

HINTS — 4 IDEAS

mean = sum(x) / n; var = sum((xi - mean)**2 for xi in x) / n (divide by n).
The denominator is sqrt(var + eps) — the eps is what saves you when var is 0.
Normalize each element: (x[i] - mean) / denom.
Then scale and shift: gamma[i] * normalized + beta[i].

CPython · WebAssembly

Approach, complexity & discussion — open after you solve

The approach

For each vector (one token’s features), compute its mean and variance over the feature dimension, subtract the mean, divide by sqrt(variance + ε), then apply the learned scale γ and shift β. The epsilon inside the square root keeps it stable when the variance is near zero.

Complexity

Time O(d) per vector in the feature dimension; space O(1) extra.

Common mistakes

Normalizing over the wrong axis — layer norm is per-example over features, unlike batch norm which normalizes over the batch.
Forgetting the epsilon, which divides by zero on a constant (zero-variance) vector.
Dropping the learned γ/β, so the layer cannot rescale or re-center after normalizing.

Where this shows up

Layer norm (and its cheaper cousin RMSNorm) stabilizes training in every Transformer, wrapped around the attention and feed-forward blocks. Because it is computed per example and not across the batch, it is independent of batch size — which is exactly why it, not batch norm, suits variable-length sequence models.

Finished this one? 0 / 75 Challenges done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering