CODING CHALLENGE · N°75

Harvest a Flaky Paginated API

Medium FDEDataReliability

The vendor API is paginated, occasionally 500s, rate-limits without documenting it, and returns overlapping pages when the data moves under you. Collect everything, retry what deserves retrying, deduplicate, and return a partial result with the failures recorded rather than an exception.

The problem

Implement harvest(fetch_page, max_attempts). fetch_page(cursor) returns a dict/object that is either {"records": [...], "next": cursor_or_None} on success or {"error": code} on failure, where code is 500, 429 or 404. Start at cursor None. Retry a page on 500 or 429 up to max_attempts total attempts; a 404 is permanent and must not be retried. Deduplicate records by their "id", keeping the first seen. If a page exhausts its attempts, stop and return what you have. Return {"records": [...], "pages": n, "failures": [...]} where pages counts pages successfully fetched and each failure is {"cursor", "code", "attempts"}.

EXAMPLE 1

Input two clean pages then next=None

Output all records, pages 2, failures []

the happy path

EXAMPLE 2

Input a 500 then success, max_attempts 3

Output records collected, no failure recorded

a retried-and-recovered page is not a failure

EXAMPLE 3

Input a 404

Output partial records + one failure, attempts 1

permanent — never retried

CONSTRAINTS

500 and 429 are retryable; 404 is permanent and consumes exactly one attempt.
Never exceed max_attempts total attempts for any single cursor.
Records are deduplicated by id, first occurrence wins.
A page that exhausts its attempts stops the crawl and returns a partial result — it never raises.

SOLVE IT YOURSELF

Your turn — write it

Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.

YOUR TASK

Implement harvest(fetch_page, max_attempts) returning records, pages and failures.

HINTS — 4 IDEAS

Two loops: an outer one over cursors, an inner one over attempts for the current cursor.
A 404 must break out of the attempt loop immediately — retrying a permanent error just burns a customer’s quota.
Keep a set of seen ids and append only unseen records, so a shifting dataset cannot duplicate rows.
On exhaustion, return what you have with the failure recorded. Never throw away collected records.

CPython · WebAssembly

Approach, complexity & discussion — open after you solve

The approach

The naive loop follows the cursor until it is null and assumes every call succeeds. Real vendor APIs fail three ways at once: transient 5xx, rate-limit 429s, and duplicate records across page boundaries when the underlying data shifts mid-crawl. Each needs a different response, and conflating them is the bug.

Retry the transient failures with a bounded attempt count. Treat a 429 as a retry too, but never as a failure to give up on. Deduplicate by record id as you collect, because a moving dataset genuinely does return the same row on two pages. And when a page exhausts its retries, do not throw away everything you already have — return the partial result with the failure recorded, because a partial harvest you know the shape of is far more useful than an exception.

Complexity

O(pages × attempts) calls; O(records) space for the dedup set.

Common mistakes

Treating a 429 as a permanent failure and abandoning the crawl — it is the one error the API is explicitly telling you to retry.
Retrying forever, so a genuinely broken page turns into an infinite loop against a customer’s rate-limited quota.
Discarding everything on a single page failure instead of returning what was collected with the failure recorded.
Not deduplicating, so a dataset that shifts mid-crawl silently produces duplicate rows downstream.

Where this shows up

Every integration meets this in week two: the customer’s vendor API is paginated, occasionally 500s, has a limit nobody documented, and returns overlapping pages when records are being written while you read. The version that returns a partial result plus a failure list is the one you can put in front of a customer, because it answers "what did we get?" rather than raising an exception at 3am with nothing to show.

Finished this one? 0 / 75 Challenges done

Explore the topic

See this alongside everything else on the same subject — handbooks, system designs, challenges and tools, in one place.

Forward Deployed Engineering