Harvest a Flaky Paginated API

Start from brute force

The obvious loop follows the cursor until it is None and assumes every call succeeds: fetch, extend records with page["records"], move to page["next"]. Against a clean API it is correct, with P calls for P pages and O(R) space for R records. Against this one it fails three ways. The first {"error": 500} has no "records" key, so it raises KeyError and every record already collected is lost. A 404 is treated as data instead of a permanent stop. And overlapping pages are appended twice: the happy-path example comes back as ids [1, 2, 2, 3].

The approach

Separate the three decisions the naive loop conflates: what to retry, how many times, and what to keep. The reference nests two loops. The outer while True walks cursors from None; the inner while attempts < max_attempts retries the current cursor. A response without an "error" key is the page and breaks the inner loop. An error in RETRYABLE = {500, 429} loops again, and any other code, such as 404, breaks at once, so a permanent error costs exactly one attempt.

If the inner loop ends with page is None, the reference appends {"cursor", "code": last_code, "attempts"} to failures and breaks the outer loop, returning what it has instead of raising. On success it increments pages and appends only records whose "id" is not yet in seen. The invariant: records holds each id once, in first-seen order, and nothing collected is ever discarded.

Walk through an example

With max_attempts = 3, a server that answers "p2" with a 500, then a 429, then a page that repeats id 2, and answers "p3" with a 500 every time, the crawl keeps ids 1 to 3 and records one failure.

Complexity

Each cursor costs at most A = max_attempts calls, so P pages take at most O(P·A) calls, and the work around them is O(R) for R records received, with O(1) set operations each. Space is O(R) for records and seen. The naive loop also makes P calls in O(R) space; its problem is correctness, not cost.

Edge cases to test

• A 404 on a later page stops the crawl and keeps the earlier records: pages = 1 and one failure, {"cursor": "p2", "code": 404, "attempts": 1}.
• A 429 that never clears is still a failure once the attempts run out: harvest(lambda c: {"error": 429}, 2) returns no records and {"cursor": None, "code": 429, "attempts": 2}.
• A retryable error followed by a 404 records the last code: answering the first cursor with 500 and then 404, under max_attempts = 5, gives {"code": 404, "attempts": 2}.
• A status outside the spec, such as 503, is not in RETRYABLE, so the reference treats it as permanent and stops after one attempt. Real crawlers usually retry 502, 503 and 504 as well.
• A server that keeps returning the same next cursor makes the reference loop forever. A production crawler keeps a set of cursors already fetched, or a page limit.
• max_attempts = 0 makes no calls at all and records {"cursor": None, "code": None, "attempts": 0}.

Common mistakes

• Treating a 429 as a permanent failure and abandoning the crawl — it is the one error the API is explicitly telling you to retry.
• Retrying forever, so a genuinely broken page turns into an infinite loop against a customer’s rate-limited quota.
• Discarding everything on a single page failure instead of returning what was collected with the failure recorded.
• Not deduplicating, so a dataset that shifts mid-crawl silently produces duplicate rows downstream.

Follow-ups an interviewer asks next

How should retries wait between attempts?

Use exponential backoff with jitter: sleep a random time up to base · 2^attempt, capped, so many clients recovering from one outage do not retry in lockstep. If a 429 response carries a Retry-After header, wait at least that long. The reference retries immediately, which is fine in a replay but would hammer a real rate limit.

The crawl died at page 40 of 100. How do you resume?

Checkpoint after every successful page: persist the next cursor, the page count and the seen ids, or a high-water mark if ids are ordered. On restart, load the checkpoint and continue from the saved cursor. The partial result the reference returns already says where to start: the failure’s cursor is the page to retry.

The dataset is too big to keep every id in memory. Now what?

Deduplicate in storage: write records to a table with a unique key on id and upsert, so a repeat overwrites or is ignored. If ids grow with time, a high-water mark replaces the set. A Bloom filter fits in bounded memory, but a false positive would silently drop a real record, so it can only screen ids before an exact check.

How would you make the crawl faster?

Cursor pagination is sequential, because each page names the next, so parallelism has to come from elsewhere: crawl independent partitions concurrently, such as date ranges or accounts, or use offset pagination where the API offers it and the data is stable. Every worker still shares one rate limit, so put a single limiter in front of all of them.

Where this shows up

Every integration meets this in week two: the customer’s vendor API is paginated, occasionally 500s, has a limit nobody documented, and returns overlapping pages when records are being written while you read. The version that returns a partial result plus a failure list is the one you can put in front of a customer, because it answers "what did we get?" rather than raising an exception at 3am with nothing to show.

Reference solution

RETRYABLE = {500, 429}


def harvest(fetch_page, max_attempts):
    records, seen, failures = [], set(), []
    cursor, pages = None, 0

    while True:
        attempts, page, last_code = 0, None, None
        while attempts < max_attempts:
            attempts += 1
            res = fetch_page(cursor)
            if "error" not in res:
                page = res
                break
            last_code = res["error"]
            if last_code not in RETRYABLE:      # 404 is permanent — stop trying
                break

        if page is None:
            failures.append({"cursor": cursor, "code": last_code, "attempts": attempts})
            break                               # partial result, not an exception

        pages += 1
        for r in page.get("records", []):
            if r["id"] not in seen:             # dedup: pages can overlap
                seen.add(r["id"])
                records.append(r)

        cursor = page.get("next")
        if cursor is None:
            break

    return {"records": records, "pages": pages, "failures": failures}


pages = [{"records": [{"id": 1}, {"id": 2}], "next": "p2"},
         {"records": [{"id": 2}, {"id": 3}], "next": None}]
seq = {"i": 0}


def fake(cursor):
    p = pages[seq["i"]]
    seq["i"] += 1
    return p


out = harvest(fake, 3)
print([r["id"] for r in out["records"]], out["pages"], out["failures"])
# -> [1, 2, 3] 2 []

const RETRYABLE = [500, 429];

function harvest(fetchPage: (c: any) => any, maxAttempts: number): any {
  const records: any[] = [], seen = new Set<any>(), failures: any[] = [];
  let cursor: any = null, pages = 0;

  for (;;) {
    let attempts = 0, page: any = null, lastCode: any = null;
    while (attempts < maxAttempts) {
      attempts++;
      const res = fetchPage(cursor);
      if (res.error === undefined) { page = res; break; }
      lastCode = res.error;
      if (RETRYABLE.indexOf(lastCode) === -1) break;      // 404 is permanent
    }

    if (page === null) {
      failures.push({ cursor: cursor, code: lastCode, attempts: attempts });
      break;                                              // partial result
    }

    pages++;
    for (const r of (page.records || [])) {
      if (!seen.has(r.id)) { seen.add(r.id); records.push(r); }
    }

    cursor = page.next === undefined ? null : page.next;
    if (cursor === null) break;
  }

  return { records: records, pages: pages, failures: failures };
}

let i = 0;
const fixture = [{ records: [{ id: 1 }, { id: 2 }], next: "p2" },
                 { records: [{ id: 2 }, { id: 3 }], next: null }];
const out = harvest(function () { return fixture[i++]; }, 3);
console.log(out.records.map(function (r) { return r.id; }), out.pages, out.failures);
// -> [1, 2, 3] 2 []