Token-Level F1
The metric behind QA evaluation (SQuAD and friends): how well does a predicted answer overlap a reference, as a bag of words? Compute token precision and recall, then combine them into the harmonic-mean F1 — the number that actually moves when your model gets better.
The problem
Given a prediction string and a reference string, return the token-level F1. Lowercase and split both on whitespace into tokens. Count the multiset overlap (shared tokens, respecting duplicates). Then precision = overlap / len(pred), recall = overlap / len(ref), and F1 = 2·P·R / (P + R). Return 0.0 if either side is empty or there is no overlap.
pred = "the cat sat", ref = "the cat sat"1.0pred = "the cat", ref = "the cat sat"0.8pred = "a b", ref = "c d"0.0- Case-insensitive; split on whitespace.
- Overlap is a multiset intersection — "the the" vs "the" shares only one "the".
- Return 0.0 when either string is empty or overlap is 0 (avoid dividing by zero).
Your turn — write it
Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.
Implement token_f1(prediction, reference): lowercase + split both, count multiset token overlap, then return the harmonic mean of precision and recall (0.0 on empty / no overlap).
- Lowercase both strings and
split()on whitespace into token lists. - For multiset overlap, count reference tokens, then for each prediction token consume one from that count if available.
- precision = overlap / len(pred); recall = overlap / len(ref).
- F1 = 2·P·R / (P + R). Guard the empty / zero-overlap cases by returning 0.0 first.
Approach, complexity & discussion — open after you solve
The approach
Treat the prediction and the reference as bags of tokens. Count the shared tokens (by multiplicity — the min of each token’s counts). Then precision = shared / predicted-length, recall = shared / reference-length, and F1 is their harmonic mean: 2·P·R / (P + R).
Complexity
Time O(n + m) in the two token counts; space O(distinct tokens).
Common mistakes
- Requiring exact-string match instead of token overlap — far too strict for free-form generation.
- Ignoring multiplicity (counting a token once when it appears several times).
- Dividing by zero when the prediction or reference is empty — guard those cases.
Where this shows up
Token-F1 is a standard eval metric for question answering and generation (the SQuAD-style overlap score) — a pragmatic middle ground between brittle exact-match and expensive LLM-as-judge. Knowing how it is computed is part of the scarce, in-demand skill of actually measuring AI quality.
Explore the topic
See this challenge alongside everything else on the same subject — handbooks, system designs, algorithms and tools, in one place.