Token-Level F1

Start from brute force

The obvious way to count shared tokens is to copy the reference token list and, for each prediction token, find and remove a matching copy with list.remove. Removing what was matched makes it respect duplicates, so the count is right. But each search scans the list, so it costs O(n · m) time for n prediction and m reference tokens, with O(m) space for the copy. The faster-looking len(set(pred) & set(ref)) is O(n + m) and wrong: it counts each distinct token once, so "the the cat" against "the the dog" finds one shared token instead of two and scores 0.333 instead of 0.667.

The approach

Count the reference once, then let each prediction token spend from that budget. The reference token_f1 lowercases both strings and calls split(), which splits on runs of whitespace and drops empty strings, and returns 0.0 if either side has no tokens. It builds counts, the multiplicity of every reference token, then walks pred: whenever counts.get(t, 0) > 0, it adds 1 to overlap and decrements counts[t].

The invariant is that overlap equals the sum over tokens of the smaller of the two counts seen so far, and counts never goes negative, so a token repeated in the prediction can match only as many copies as the reference holds. With overlap known, precision = overlap / len(pred), recall = overlap / len(ref), and the return value is 2 * precision * recall / (precision + recall), which simplifies to 2 · overlap / (len(pred) + len(ref)). The TypeScript tokenF1 splits with /\s+/ and filters out empty strings to match Python’s split().

Walk through an example

The code’s own example, "the cat" against "the cat sat", gives perfect precision and two-thirds recall.

Complexity

Time O(n + m) for n prediction and m reference tokens, plus the linear cost of lowercasing and splitting the strings: one pass builds counts and one pass spends from it, with average-O(1) dictionary operations. Space O(n + m) for the token lists and counts. The remove-from-a-list version is O(n · m) time with O(m) space.

Edge cases to test

• "the the" against "the" → 0.6666666666666666. The reference has one "the", so only one prediction token matches: precision 0.5, recall 1.0.
• "the the cat" against "the the dog" → 0.6666666666666666. Both copies of "the" match; a set intersection counts one and returns 0.333.
• "" or " " against "the cat" → 0.0. split() yields no tokens, and the guard returns before any division.
• "The CAT" against "the cat" → 1.0, and "cat the" against "the cat" → 1.0. Case is folded and order is ignored: the tokens are a bag.
• "cat." against "cat" → 0.0. Punctuation stays attached to the token here; the official SQuAD script strips punctuation before comparing.
• "not guilty" against "guilty" → 0.6666666666666666. Opposite meaning, decent score: token overlap knows nothing about negation.

Common mistakes

• Requiring exact-string match instead of token overlap — far too strict for free-form generation.
• Ignoring multiplicity (counting a token once when it appears several times).
• Dividing by zero when the prediction or reference is empty — guard those cases.

Follow-ups an interviewer asks next

Match the official SQuAD evaluation script.

Normalize before tokenizing: lowercase, strip punctuation, remove the articles a, an and the, and collapse whitespace, then compute the same F1. Normalization changes scores: "the cat" against "the cat sat" becomes "cat" against "cat sat", which scores 0.667 instead of 0.8. Report which normalization you used, or numbers are not comparable.

Each question has several acceptable reference answers.

Score the prediction against every reference and take the maximum, which is what SQuAD does. Averaging would penalize a correct answer for not also matching the other phrasings. The dataset score is then the mean of those per-question maxima, so every question counts equally however long its answer is.

When is token F1 the wrong metric?

When meaning depends on more than shared words. It ignores order and negation, so "not guilty" scores 0.667 against "guilty", and a correct paraphrase with no words in common scores 0. Report exact match alongside it for short factual answers, and for free-form generation use embedding similarity or a calibrated LLM judge, spot-checked by people.

Write the overlap count in one line.

In Python, collections.Counter supports multiset intersection: sum((Counter(pred) & Counter(ref)).values()) keeps, for each token, the smaller of its two counts. That is exactly the count the reference reaches by decrementing, so the F1 is unchanged, and the whole function stays O(len(pred) + len(ref)). For "the cat sat on the mat" against "the cat" the intersection holds one the and one cat, an overlap of 2.

Where this shows up

Token-F1 is a standard eval metric for question answering and generation (the SQuAD-style overlap score) — a pragmatic middle ground between brittle exact-match and expensive LLM-as-judge. Knowing how it is computed is part of the scarce, in-demand skill of actually measuring AI quality.

Its relatives. Token F1 is a unigram overlap score, and the same counting-with-a-budget idea is the core of the older text-generation metrics. BLEU, from machine translation, is built on precision: it clips each n-gram’s count by its count in the references, exactly the decrement here, for n-grams up to length 4, and adds a brevity penalty so that very short outputs cannot win on precision alone. ROUGE, from summarization, is built on recall of the reference’s n-grams. Moving from single tokens to bigrams is how these metrics start to notice word order, which token F1 ignores completely.

Empty answers are a policy decision. This reference returns 0.0 whenever either side has no tokens, including when both are empty. Benchmarks with unanswerable questions, where the correct output is “no answer”, need the opposite for that case: the official SQuAD 2.0 evaluation scores a pair as 1 if both the prediction and the reference are empty and 0 if only one is. If your dataset includes questions with no answer, this one guard decides whether a model is rewarded or punished for correctly declining to answer.

Use it with care in LLM evaluations. Chat models answer in full sentences, so against a short reference such as "Paris" a correct answer like "The capital is Paris." loses heavily on precision, and its trailing period keeps "paris." from matching at all without the SQuAD normalization. Teams therefore either prompt for short answers, extract the answer span before scoring, or report recall alongside F1, since recall asks only whether the reference tokens appear in the answer.

Check yourself

Reference solution

def token_f1(prediction, reference):
    pred = prediction.lower().split()
    ref = reference.lower().split()
    if not pred or not ref:
        return 0.0

    # multiset overlap: consume each matched reference token once
    counts = {}
    for t in ref:
        counts[t] = counts.get(t, 0) + 1
    overlap = 0
    for t in pred:
        if counts.get(t, 0) > 0:
            overlap += 1
            counts[t] -= 1
    if overlap == 0:
        return 0.0

    precision = overlap / len(pred)
    recall = overlap / len(ref)
    return 2 * precision * recall / (precision + recall)


print(token_f1("the cat", "the cat sat"))   # -> 0.8

function tokenF1(prediction: string, reference: string): number {
  const pred = prediction.toLowerCase().split(/\s+/).filter(Boolean);
  const ref = reference.toLowerCase().split(/\s+/).filter(Boolean);
  if (pred.length === 0 || ref.length === 0) return 0.0;

  // multiset overlap: consume each matched reference token once
  const counts = new Map<string, number>();
  for (const t of ref) counts.set(t, (counts.get(t) || 0) + 1);
  let overlap = 0;
  for (const t of pred) {
    if ((counts.get(t) || 0) > 0) { overlap++; counts.set(t, counts.get(t)! - 1); }
  }
  if (overlap === 0) return 0.0;

  const precision = overlap / pred.length;
  const recall = overlap / ref.length;
  return (2 * precision * recall) / (precision + recall);
}

console.log(tokenF1("the cat", "the cat sat"));   // -> 0.8