CODING CHALLENGE · N°12

TF-IDF

Medium AI EngineeringNLPRetrieval

The scoring that ran search for decades — and still seeds hybrid retrieval today. It rewards a word that is frequent in one document but rare across the corpus, and shrugs off words that appear everywhere. No numpy, just a logarithm.

The problem

Implement TF-IDF: the weight of a term in one document of a corpus. The corpus is a list of documents, each a list of tokens; doc_index selects the document. Term frequency is the term's share of that document: tf = count(term in doc) / len(doc). Inverse document frequency uses how many documents contain the term (df) out of N total: idf = log(N / df). Return tf · idf. If the term is absent from the document, its weight is 0.

EXAMPLE 1

Input corpus=[['a','b'],['a','c'],['a','d']], doc_index=0, term='a'

Output 0.0

'a' is in every doc → idf = log(1) = 0 → weight 0

EXAMPLE 2

Input corpus=[['a','a','b','c'],['d'],['e']], doc_index=0, term='a'

Output ≈0.5493

tf = 2/4, idf = log(3) → a rare, frequent term scores high

EXAMPLE 3

Input corpus=[['a','b'],['c']], doc_index=0, term='z'

Output 0.0

term not in the document → weight 0

CONSTRAINTS

Term frequency: tf = count(term in doc) / len(doc).
Document frequency df = number of documents containing the term; idf = log(N / df) with N = len(corpus) (natural log).
If the term is absent from the document, return 0 (so you never divide by df = 0).
Pyodide has no numpy — use the math stdlib and plain lists only.

SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement tf_idf(corpus, doc_index, term). Return tf · idf for the term in the selected document — the term's share of that doc, scaled by how rare it is across the corpus.

HINTS — 4 IDEAS

tf = (times the term appears in the doc) / (number of tokens in the doc).
If tf is 0 the whole weight is 0 — return early, and you avoid a df = 0 divide.
df = how many documents contain the term at all; idf = log(N / df) with N = len(corpus).
A term in every document has df = N, so idf = log(1) = 0 — it carries no weight.

CPython · WebAssembly

Approach, complexity & discussion — open after you solve

The approach

Score a term in a document as TF × IDF: term frequency (how often it appears in the document) times inverse document frequency (log(total docs / docs containing the term)). TF rewards terms central to a document; IDF discounts terms that appear everywhere, so a rare, document-specific word scores high and a stopword scores near zero.

Complexity

Time O(total tokens) to build the counts; O(query terms × matching docs) to score a query.

Common mistakes

Using raw term frequency without IDF — common words then dominate every document.
Mishandling IDF edge cases: a term in every document has IDF 0; smoothing avoids division-by-zero.
Skipping document-length normalization, so long documents score artificially high.

Where this shows up

TF-IDF is the classic lexical weighting that BM25 refines with saturation and length normalization, and it is still the baseline for keyword search, document-similarity, and as features for text classification. It is the “rare words carry the signal” intuition made quantitative.

Finished this one? 0 / 75 Challenges done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

RAG & Retrieval LLM Engineering