The whole design, in writing
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a web crawler?
A crawler downloads the web — start from a few URLs, fetch each page, find its links, fetch those too, forever. Trivial as a weekend script; brutal at billions of pages, where you must be fast, polite, dedup-aware, and never get stuck in a loop or a trap.
Model it as one big loop over a queue: pull a URL, fetch it, extract links, enqueue the new ones. Then make that loop scale, behave, and stay efficient. Everything we add defends that simple loop against the messy reality of the web.
What the new pieces do
- Seed URLsinput
- The initial list of URLs to crawl from — a few well-connected pages from which the rest of the reachable web unfolds, link by link.
Step 1 · The loop
Fetch, parse, enqueue
At its heart a crawl is a graph traversal: you have URLs to visit, you visit one, it reveals more URLs, repeat. You need somewhere to hold the “to-visit” list and workers to do the visiting.
A crawl reveals more URLs as it fetches. How do you structure the loop?
Deep recursion blows the stack, can’t be distributed, and couples fetch to parse so neither scales. The to-visit set needs to be explicit state, not the call stack.
A single sequential worker can’t approach billions of pages, and mixes network-bound fetching with CPU-bound parsing. You need a shared queue and many workers.
The frontier holds what to crawl next; fetchers download, the parser extracts links and pushes new ones back. Decoupling fetch (network-bound) from parse (CPU-bound) lets each scale on its own.
A URL Frontier holds what to crawl next. Fetcher workers pull from it and download pages; the Parser extracts links and pushes the new ones back into the frontier. Seeds prime the loop and it runs itself from there.
Why this piece earns its place
The piece this step quietly assumes is that a URL leaves the frontier and comes back as links. Between those two moments it lives nowhere durable, and a fetcher can die in that gap — mid-download, or after downloading but before the parser has extracted anything. The URL is simply gone, and nothing notices: no error, no retry, just a page the crawl never visits. You cannot recover it by rediscovery either, because the seen-check added next marks a URL the moment it is enqueued, so the next link pointing at it is dropped as already-handled. So the frontier has to be a queue with leases and acknowledgements, not a pop: a worker takes a lease on a URL, and the URL only really leaves when the parser has committed its output. A crashed worker’s lease expires and the URL is redelivered. That makes delivery at-least-once rather than at-most-once, which is the right way round here — fetching a page twice costs one extra request, losing it costs it forever.
What the new pieces do
- URL Frontierbackend
- The brain of the crawler: the queue of URLs waiting to be fetched. Its ordering decides what gets crawled, how soon, and how often.
- Fetcher Workersworker
- Pull URLs from the frontier and download the HTML over HTTP. A large, mostly-waiting-on-the-network fleet — the crawler’s muscle.
- Parserworker
- Reads each downloaded page, pulls out the links, normalizes them, and feeds the new ones back into the frontier — that loop is the crawl.
Step 2 · Don’t crawl forever
Have I seen this URL?
The web is a cyclic graph — pages link back and forth endlessly. Without memory, the crawler re-queues the same URLs over and over and never makes progress, drowning in duplicates.
The web is cyclic — pages link back and forth. How do you avoid re-crawling forever, at billions of URLs?
An exact set of billions of URLs is hundreds of GB — too big for memory on every frontier node. You need a far more compact "have I seen this?" structure.
A bloom filter answers "probably seen / definitely not" in O(1) using a few bits per URL. It may rarely skip a new URL but never re-crawls a seen one — "mostly right and tiny" beats "exact and enormous".
Without an up-front seen-check the crawler loops on cycles and never makes forward progress — it drowns in duplicates before any dedup stage runs. Termination has to happen at enqueue time.
Before enqueuing a link, check a Seen Set. At billions of URLs, an exact set is too big for memory, so use a bloom filter: O(1) “probably seen / definitely not seen” in a fraction of the space, backed by a durable set for certainty.
Why this piece earns its place
The filter comes with a dial — bits per URL, which sets the false-positive rate — and both directions of getting it wrong cost something specific. Too few bits and the rate climbs, and every false positive is a real page the crawler silently declines to visit. Nothing logs it; coverage just quietly erodes while every dashboard looks healthy. Too many bits and you are walking back toward the exact set you rejected for being too big to hold per node. What makes the choice harder is that a bloom filter has no delete and its error rate is a function of how full it is, so it degrades as the crawl succeeds: sized for the URLs you have today, it is wrong by the time you have billions. Size it for the count you intend to reach, not the one you are at, and plan to roll onto a fresh, larger generation. Where coverage matters more than memory, confirm a positive against the durable set behind it — that turns a false positive into an extra lookup rather than a lost page, at the cost of a disk read on every hit.
- billionsURLs to track
- O(1)seen check
- ~bitsper URL
What the new pieces do
- Seen Setstore
- Remembers which URLs have already been queued so the crawler doesn’t loop forever. A bloom filter answers “seen this?” in O(1) using tiny memory.
Back of the envelope
- billions of URLs, exact set ⇒ 100s of GB
- too big to hold in memory per node
- bloom filter ⇒ ~a few bits / URL
- O(1) check in a fraction of the space
- rare false positive
- may skip a new URL, never re-crawls a seen one
Step 3 · Be a good citizen
Politeness & robots.txt
Thousands of fetchers all hammering one small site at once is indistinguishable from an attack — you’ll knock it over and get your crawler banned. And many sites declare what you may and may not crawl.
Thousands of fetchers could hammer one small site at once. How do you not look like an attack?
A global cap still lets all that traffic pile onto one unlucky host while big sites are under-crawled. Politeness is a per-domain promise, not a global throttle.
Maximum speed per host is exactly the DDoS that knocks small servers over and gets your IPs banned — ending your crawl. Speed must be bounded per host.
Obey each site’s rules and pace requests per domain, while crawling thousands of hosts simultaneously. Gentle on each, fast in aggregate — and the reason hosts become the partition unit later.
Add a Politeness layer: honor each host’s robots.txt, and enforce a per-host rate limit with a delay between requests to the same domain. Group the frontier by host so one site’s crawl is paced independently of others.
Why this piece earns its place
Politeness quietly changes what limits throughput. Before this step the crawl rate was a function of how many fetchers you ran; after it, it is a function of how many distinct hosts are ready to be fetched right now. If the frontier hands out URLs in discovery order, a worker that draws a URL from a host it just hit has to wait out that host’s delay, and the fleet ends up idling behind a handful of popular domains while millions of URLs on other hosts sit unfetched. So the frontier stops being one queue: keep a back-queue per host and a schedule keyed by each host’s next-allowed time, and only ever hand a worker a host whose delay has already elapsed. Fetchers should never block — they should only be given work that is legal to do. The other thing to check is that a host is not a machine. Plenty of hostnames resolve to one shared server, so a per-host promise kept perfectly can still land as one pile of traffic on it; pacing by resolved IP as well as by name is what makes the promise real.
- robots.txtobeyed
- 1/hostrate limited
- parallelacross hosts
What the new pieces do
- Politenessguard
- Obeys each site’s robots.txt and caps requests per host, so the crawler is a good citizen and doesn’t accidentally DDoS a small server.
Step 4 · The hidden bottleneck
DNS at scale
Every fetch needs a hostname resolved to an IP. DNS lookups are slow (tens of ms) and, at billions of requests, become the dominant cost — and can hammer DNS servers as hard as you hammer websites.
Every fetch resolves a hostname to an IP, and DNS is slow (tens of ms). At billions of fetches, what breaks?
At billions of requests, tens-of-ms lookups become the dominant cost and can hammer DNS servers as hard as you hammer sites. Ignoring DNS makes it the silent bottleneck.
Most pages on a site share one IP, so the first lookup serves thousands of fetches; async resolution keeps fetchers from blocking. The unglamorous infra often decides throughput more than scheduling.
IPs change and rotate (CDNs, load balancers), so hard-coding breaks constantly and silently fetches the wrong server. You need caching with a TTL, not static mappings.
Put a DNS Cache in front of resolution and resolve asynchronously so fetchers don’t block. Cache by host with a TTL; most pages on a site share one IP, so the first lookup serves thousands of fetches.
Why this piece earns its place
The TTL on this cache is the knob, and the honest answer is that you cannot simply honor the record’s own. Large sites sit behind CDNs and publish very short TTLs precisely so they can move traffic, and at this fetch volume that means re-resolving the hottest hosts constantly — the cache stops being a cache exactly where it would have paid best. Override it upward and you get the opposite failure: you keep connecting to an address the site has moved off, and that does not arrive as a clean error, it arrives as refused connections or as somebody else’s server answering on a recycled address. The workable setting is a floor and a ceiling of your own over whatever the record says, with a shorter ceiling for hosts that have already shown you they move. Two things are easy to leave out. Cache the failures too, or a host that does not resolve gets retried once for every URL of its queued behind it. And resolve a given hostname one request at a time — when a popular host’s entry expires, every fetcher holding one of its URLs misses in the same instant and they all ask at once.
What the new pieces do
- DNS Cacheservice
- Resolving hostnames is slow and would dominate latency at billions of fetches. A cache (and async resolver) keeps lookups from becoming the bottleneck.
Back of the envelope
- lookup ≈ tens of ms
- dominant per-fetch cost at billions of fetches
- 1 host ⇒ ~1 IP
- first lookup serves thousands of fetches
- cache + async resolve
- keeps the fetch pipeline full, fetchers unblocked
Step 5 · Keep what you fetched
Store & dedup content
Downloaded pages must be stored for downstream indexing — but the web is full of duplicate content: mirrors, print versions, session-ID URLs. Different URLs, identical pages. Storing them all wastes enormous space and indexing effort.
Mirrors, print versions and session-ID URLs deliver identical pages under different URLs. How do you not store the web twice?
The bloom filter dedups URLs, not content — a brand-new URL can still return a page you’ve stored a thousand times. URL-dedup and content-dedup are different jobs.
Comparing against the entire corpus per page is impossibly expensive. You need a compact fingerprint to check, not a full-content comparison.
A content hash (or near-dup signature like SimHash) checked against seen Content Hashes catches mirrors and boilerplate that are different URLs but the same page — saving storage and indexing effort.
Write raw pages to a Content Store (blob storage). Compute a content fingerprint (a hash, or a near-dup signature like SimHash) and check it against seen Content Hashes — skip pages whose content you already have under another URL.
Why this piece earns its place
A plain hash of the bytes is weaker than it sounds. Rotate an ad, stamp a render time into the footer, bump a visitor counter, and a perfect mirror hashes to something new — the pages you most want to catch are the ones that differ by a handful of bytes. That is what the near-duplicate signature is for, and it brings a threshold with it: a similarity above which two pages count as the same. Push it too far and you lose real pages in a specific, damaging shape, because sites built from one template share nav, sidebar and footer. Two genuinely different product pages can look near-identical to a naive signature while the part that actually differs — a few lines of body text — is swamped by the boilerplate around it. So fingerprint the extracted main content rather than the raw document, and set the threshold against pages you already know are distinct instead of picking a number that sounds careful. And be precise about what this buys: the fetch has already happened. Content dedup saves storage and indexing work, not bandwidth.
What the new pieces do
- Content Storestore
- Durable storage (blob store) for the fetched pages — the actual payload downstream indexers and analyzers consume.
- Content Hashesstore
- Fingerprints of page content. Catches mirror sites and boilerplate that are different URLs but the same page, so you don’t store the web twice.
Step 6 · Not all pages are equal
Prioritize the frontier
Pure BFS treats a spam page like the homepage of a major news site, and never decides when to re-crawl a page that changes hourly versus one that never changes. Crawl budget is finite; spending it evenly is wrong.
Make the frontier a priority queue: rank URLs by importance (PageRank-ish signals, depth) and schedule re-crawls by how often a page actually changes. Important, fast-changing pages get crawled sooner and more often.
Why this piece earns its place
Priority introduces starvation, and a re-crawl schedule that learns from observation turns starvation into a loop it cannot climb out of. A page’s change-rate estimate only improves when you fetch it again; a page ranked low is never fetched again; so its estimate never moves, and the page stays low on evidence you stopped collecting. The same trap sits on the importance side, because link-based signals only see pages you have crawled — a poorly linked corner of the web is invisible because it is uncrawled, and uncrawled because it looks unimportant. Two defenses. Put age into the priority so anything waiting long enough eventually rises whatever its score, and reserve a slice of crawl budget for exploration rather than giving all of it to the top of the queue. Then make re-crawls cheap enough that you can afford to be wrong about them: a conditional request carrying the stored validator lets a server answer not modified without sending the body, so you can probe a page’s change rate far more often than you could afford to download it.
Step 7 · Scale & traps
Shard out, avoid traps
One machine can’t hold the frontier or fetch the web. And the web is hostile: crawler traps (infinite calendars, endless query-param permutations) can trap a naive crawler forever in one site.
One machine can’t hold the frontier or fetch the whole web. How do you shard it while keeping politeness?
Random sharding scatters one host’s URLs across many nodes, so no single node can enforce that host’s rate limit — politeness breaks. The shard key has to keep a host together.
Hashing the host to a shard puts all of a site’s URLs — and its rate limit — on one node, so per-host politeness stays local and correct. Add nodes to crawl more hosts in parallel.
Full replication means every node sees every URL and they’d duplicate work or need constant coordination to divide it. You want to partition the frontier, not copy it everywhere.
Shard the frontier by host across many machines so each shard owns some domains (keeping politeness local) and the fleet scales horizontally. Defend against traps with depth limits, URL-length caps, parameter normalization and per-host page budgets.
Why this piece earns its place
Sharding by host buys politeness; it does not buy balance, and it is worth saying so before someone measures it. Hosts are wildly unequal — a few sites hold an enormous share of the web’s pages while most hold a handful — so hashing hostnames spreads hosts evenly across nodes and URLs not at all. One shard draws a giant site and becomes the tail that decides when a crawl round ends, and it cannot work its way out, because the per-host rate limit from the politeness step caps exactly the queue that is backed up. Two things follow. Make shards far more numerous than machines and keep the shard-to-machine mapping a lookup table rather than a modulo, so a heavy shard can be moved without rehashing the world — its hosts’ back-queues and seen-set slice travel with it. And treat the per-host page budget introduced here as the real bound rather than a trap-only defense: it is what stops any one host, hostile or merely enormous, from owning a shard indefinitely.
What the new pieces do
- Sharded Frontierbus
- The frontier spread across many machines, partitioned by host so per-host politeness is local and the whole thing scales horizontally.
Back of the envelope
- partition by host (hash)
- a site’s URLs + rate limit live on one node
- add nodes ⇒ more hosts in parallel
- horizontal scale, politeness preserved
- depth / param / length caps
- defuse infinite calendars and query-param traps
You did it
You just designed a web crawler.
Everything you assembled, in order
- A URL frontier feeding fetch → parse → enqueue: the core crawl loop.
- A bloom-filter Seen Set stops infinite re-crawling of the cyclic web graph.
- Politeness: obey robots.txt and rate-limit per host so you’re not an attack.
- A DNS cache + async resolution removes the hidden per-fetch bottleneck.
- A content store plus content-hash dedup avoids storing mirrors and boilerplate twice.
- A priority frontier balances importance and re-crawl freshness, not blind FIFO.
- Shard the frontier by host to scale out, with depth/param limits to dodge traps.