Harvest a Flaky Paginated API
The vendor API is paginated, occasionally 500s, rate-limits without documenting it, and returns overlapping pages when the data moves under you. Collect everything, retry what deserves retrying, deduplicate, and return a partial result with the failures recorded rather than an exception.
The problem
Implement harvest(fetch_page, max_attempts). fetch_page(cursor) returns a dict/object that is either {"records": [...], "next": cursor_or_None} on success or {"error": code} on failure, where code is 500, 429 or 404. Start at cursor None. Retry a page on 500 or 429 up to max_attempts total attempts; a 404 is permanent and must not be retried. Deduplicate records by their "id", keeping the first seen. If a page exhausts its attempts, stop and return what you have. Return {"records": [...], "pages": n, "failures": [...]} where pages counts pages successfully fetched and each failure is {"cursor", "code", "attempts"}.
two clean pages then next=Noneall records, pages 2, failures []a 500 then success, max_attempts 3records collected, no failure recordeda 404partial records + one failure, attempts 1500and429are retryable;404is permanent and consumes exactly one attempt.- Never exceed
max_attemptstotal attempts for any single cursor. - Records are deduplicated by id, first occurrence wins.
- A page that exhausts its attempts stops the crawl and returns a partial result — it never raises.
Your turn — write it
Edit the stub, hit Run (or ⌘/Ctrl + Enter), and watch the hidden tests. Stuck? the hints are right above and Reveal solution is one click away.
Implement harvest(fetch_page, max_attempts) returning records, pages and failures.
- Two loops: an outer one over cursors, an inner one over attempts for the current cursor.
- A 404 must break out of the attempt loop immediately — retrying a permanent error just burns a customer’s quota.
- Keep a set of seen ids and append only unseen records, so a shifting dataset cannot duplicate rows.
- On exhaustion, return what you have with the failure recorded. Never throw away collected records.
Approach, complexity & discussion — open after you solve
The approach
The naive loop follows the cursor until it is null and assumes every call succeeds. Real vendor APIs fail three ways at once: transient 5xx, rate-limit 429s, and duplicate records across page boundaries when the underlying data shifts mid-crawl. Each needs a different response, and conflating them is the bug.
Retry the transient failures with a bounded attempt count. Treat a 429 as a retry too, but never as a failure to give up on. Deduplicate by record id as you collect, because a moving dataset genuinely does return the same row on two pages. And when a page exhausts its retries, do not throw away everything you already have — return the partial result with the failure recorded, because a partial harvest you know the shape of is far more useful than an exception.
Complexity
O(pages × attempts) calls; O(records) space for the dedup set.
Common mistakes
- Treating a 429 as a permanent failure and abandoning the crawl — it is the one error the API is explicitly telling you to retry.
- Retrying forever, so a genuinely broken page turns into an infinite loop against a customer’s rate-limited quota.
- Discarding everything on a single page failure instead of returning what was collected with the failure recorded.
- Not deduplicating, so a dataset that shifts mid-crawl silently produces duplicate rows downstream.
Where this shows up
Every integration meets this in week two: the customer’s vendor API is paginated, occasionally 500s, has a limit nobody documented, and returns overlapping pages when records are being written while you read. The version that returns a partial result plus a failure list is the one you can put in front of a customer, because it answers "what did we get?" rather than raising an exception at 3am with nothing to show.
Explore the topic
See this challenge alongside everything else on the same subject — handbooks, system designs, algorithms and tools, in one place.