Cross-Entropy Loss

Start from brute force

The obvious approach turns an integer target into a one-hot vector and always evaluates the full formula, −Σ tᵢ·log(pᵢ), over all K classes. It is correct in exact arithmetic, because every term with tᵢ = 0 is zero. It costs O(K) time and O(K) space for the one-hot vector, for an answer that depends on one number. Unguarded, it is also wrong on valid inputs: any class with probability 0 makes it take log(0), even a class whose weight is 0. Python’s math.log(0.0) raises ValueError, and JavaScript computes 0 * Math.log(0) as NaN.

The approach

With a one-hot target, only the true class survives the sum, so the loss is a single term: −log(p[target]). The reference branches on the target’s type. For an integer it returns -math.log(max(p[target], eps)), which is O(1). For a list it keeps the general sum, -sum(t * math.log(max(pi, eps)) for t, pi in zip(target, p)), which also accepts soft labels such as label-smoothed targets.

In both branches max(..., eps) clamps each probability to at least 1e-12 before the log, so a zero probability yields a large finite loss of about 27.63 instead of an error, and the clamp is applied term by term, so a zero in a weight-0 class is harmless too. Since every probability is at most 1, the loss is never negative, and it is 0 only when the true class gets probability 1.

Walk through an example

With p = [0.5, 0.0, 0.25, 0.25] and the one-hot target = [0, 0, 1, 0], the clamp keeps the zero-probability class 1 from crashing the sum, and only class 2 contributes, giving log 4 ≈ 1.3863.

Complexity

The integer form is O(1) time and space: one lookup and one log. The list form is O(K) time for K classes and O(1) extra space, since the sum runs over a generator. Building a one-hot vector for every integer target costs O(K) time and O(K) space per example, so a batch of n examples costs O(n·K) instead of O(n).

Edge cases to test

• Zero probability on the true class: cross_entropy([1.0, 0.0], 1) returns 27.631…, which is −log(1e-12). Unclamped, Python’s math.log(0.0) raises ValueError: math domain error, and JavaScript’s Math.log(0) returns -Infinity.
• Zero probability on a weight-0 class in the list form: cross_entropy([0.0, 0.5, 0.5], [0, 1, 0]) returns 0.6931…. Unclamped, Python raises and JavaScript returns NaN, because 0 * -Infinity is NaN.
• A perfect prediction returns -0.0 in Python and -0 in TypeScript, because negating log(1) = 0.0 flips the sign bit. It compares equal to 0; only the printed sign differs.
• The target’s type picks the formula: a tuple such as (0, 1) is not a list, so the reference uses it as an index and raises TypeError. A negative index passes silently in Python (target = -1 reads the last class), while in TypeScript p[-1] is undefined and the loss is NaN.
• Soft labels go through the list branch: cross_entropy([0.7, 0.2, 0.1], [0.9, 0.05, 0.05]) returns 0.5166….

Common mistakes

• Summing −log(p) over the whole distribution instead of taking only the target class.
• Dropping the negative sign — loss must be non-negative and grow as the model gets it wrong.
• Taking log(0) when the target probability underflowed to exactly 0 — clip with an epsilon.

Follow-ups an interviewer asks next

The model outputs logits, not probabilities. How do you compute the loss stably?

Skip the probabilities: loss = logsumexp(z) − z[target], with logsumexp(z) = m + log Σ exp(zᵢ − m) and m = max(z). For z = [1000.0, 0.0] and target 1 this gives exactly 1000.0, while softmax-then-log underflows exp(−1000) to 0.0 and reports the clamp’s 27.63. PyTorch’s CrossEntropyLoss takes raw logits for this reason.

How do you compute it over a batch?

Average the per-example losses. For the examples ([0.7,0.2,0.1], 0), ([0.1,0.8,0.1], 1) and ([0.3,0.3,0.4], 0) the losses are about 0.357, 0.223 and 1.204, and the mean is about 0.595. The mean keeps the gradient scale independent of batch size. Language models average over every real token and mask out padding positions.

What about binary classification with a single probability?

Use binary cross-entropy: −(y·log p + (1 − y)·log(1 − p)) for a label y of 0 or 1. It is this function’s two-class case with the distribution [1 − p, p]. Clamp p into [eps, 1 − eps] so neither logarithm ever sees 0.

Where this shows up

Cross-entropy is the loss that trains almost every classifier and language model: for an LM, the target is the actual next token and the loss is −log of the probability the model assigned it. Because the penalty explodes as that probability approaches 0, a single confidently-wrong label dominates the average — which is why label noise is so damaging and why models learn hard to never be certain and wrong.

Reference solution

import math

def cross_entropy(p, target, eps=1e-12):
    if isinstance(target, list):
        # one-hot target: -sum(t * log(p))
        return -sum(t * math.log(max(pi, eps)) for t, pi in zip(target, p))
    # integer class index
    return -math.log(max(p[target], eps))


print(cross_entropy([0.0, 1.0, 0.0], 1))                 # -> 0.0
print(cross_entropy([0.25, 0.25, 0.25, 0.25], 0))        # ~1.3863 (= log 4)
print(cross_entropy([0.999, 0.0005, 0.0005], 1))         # ~7.6

function crossEntropy(p: number[], target: number | number[], eps = 1e-12): number {
  if (Array.isArray(target)) {
    // one-hot target: -sum(t * log(p))
    return -target.reduce((acc, t, i) => acc + t * Math.log(Math.max(p[i], eps)), 0);
  }
  // integer class index
  return -Math.log(Math.max(p[target], eps));
}

console.log(crossEntropy([0.0, 1.0, 0.0], 1));          // -> 0
console.log(crossEntropy([0.25, 0.25, 0.25, 0.25], 0)); // ~1.3863 (= log 4)
console.log(crossEntropy([0.999, 0.0005, 0.0005], 1));  // ~7.6