CODING CHALLENGE · N°10

Cross-Entropy Loss

Easy AI EngineeringDeep LearningLoss Functions

The loss that trains almost every classifier and language model. It measures how surprised the model was by the right answer: if it put high probability on the true class, the loss is near 0; if it was confidently wrong, the loss explodes. No numpy — just a logarithm.

The problem

Implement cross-entropy loss for a single example. You are given a predicted probability distribution p (a list that sums to ~1) and a target. The target is either an integer class index or a one-hot list. Return the negative log-probability the model assigned to the true class: -log(p[target]) for an index, or -Σ tᵢ·log(pᵢ) for a one-hot target. Guard the logarithm with a tiny eps so a probability of 0 gives a large finite loss instead of -inf.

EXAMPLE 1

Input p=[0.0, 1.0, 0.0], target=1

Output 0.0

all probability on the right class → no surprise, no loss

EXAMPLE 2

Input p=[0.25, 0.25, 0.25, 0.25], target=0

Output ≈1.3863

a uniform guess over K=4 → loss = log(4)

EXAMPLE 3

Input p=[0.999, 0.0005, 0.0005], target=1

Output ≈7.6

confident and wrong → large loss

CONSTRAINTS

For an integer target, the loss is -log(p[target]).
For a one-hot list target, the loss is -Σ tᵢ·log(pᵢ).
Clamp each probability to at least eps = 1e-12 before taking its log, so log(0) never happens.
Pyodide has no numpy — use the math stdlib and plain lists only.

SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement cross_entropy(p, target, eps=1e-12). Return the negative log-probability of the true class — accepting either an integer index or a one-hot list as the target.

HINTS — 4 IDEAS

The whole loss for an integer target is one line: -log(p[target]).
Protect the log with max(p[target], eps) so a zero probability gives a big finite number, not -inf.
If the target is a list (one-hot), sum -tᵢ·log(pᵢ) over all classes — only the tᵢ=1 term survives.
Perfect confidence (p on the true class = 1) gives log(1) = 0; a uniform guess over K classes gives log(K).

CPython · WebAssembly

Approach, complexity & discussion — open after you solve

The approach

Cross-entropy for one example is just the negative log of the probability the model gave the true class: −log(p[target]). You do not need the whole distribution — only the target’s probability. Clip it away from exactly 0 first (e.g. a tiny epsilon) so a confidently-wrong prediction gives a large finite loss instead of −log(0) = ∞.

Complexity

Time O(1) per example (O(n) over a batch of n); space O(1).

Common mistakes

Summing −log(p) over the whole distribution instead of taking only the target class.
Dropping the negative sign — loss must be non-negative and grow as the model gets it wrong.
Taking log(0) when the target probability underflowed to exactly 0 — clip with an epsilon.

Where this shows up

Cross-entropy is the loss that trains almost every classifier and language model: for an LM, the target is the actual next token and the loss is −log of the probability the model assigned it. Because the penalty explodes as that probability approaches 0, a single confidently-wrong label dominates the average — which is why label noise is so damaging and why models learn hard to never be certain and wrong.

Finished this one? 0 / 75 Challenges done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

LLM Engineering