The whole design, in writing
Learn system design by building a distributed web crawler step by step. An interactive guide covering the URL frontier, the fetch–parse–enqueue loop, dedup with bloom filters, politeness and robots.txt, DNS caching, content storage and dedup, and sharding the frontier.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What is a web crawler?
A crawler downloads the web — start from a few URLs, fetch each page, find its links, fetch those too, forever. Trivial as a weekend script; brutal at billions of pages, where you must be fast, polite, dedup-aware, and never get stuck in a loop or a trap.
Model it as one big loop over a queue: pull a URL, fetch it, extract links, enqueue the new ones. Then make that loop scale, behave, and stay efficient. Everything we add defends that simple loop against the messy reality of the web.
What the new pieces do
- Seed URLsinput
- The initial list of URLs to crawl from — a few well-connected pages from which the rest of the reachable web unfolds, link by link.
Step 1 · The loop
Fetch, parse, enqueue
At its heart a crawl is a graph traversal: you have URLs to visit, you visit one, it reveals more URLs, repeat. You need somewhere to hold the “to-visit” list and workers to do the visiting.
A crawl reveals more URLs as it fetches. How do you structure the loop?
Deep recursion blows the stack, can’t be distributed, and couples fetch to parse so neither scales. The to-visit set needs to be explicit state, not the call stack.
A single sequential worker can’t approach billions of pages, and mixes network-bound fetching with CPU-bound parsing. You need a shared queue and many workers.
The frontier holds what to crawl next; fetchers download, the parser extracts links and pushes new ones back. Decoupling fetch (network-bound) from parse (CPU-bound) lets each scale on its own.
A URL Frontier holds what to crawl next. Fetcher workers pull from it and download pages; the Parser extracts links and pushes the new ones back into the frontier. Seeds prime the loop and it runs itself from there.
What the new pieces do
- URL Frontierbackend
- The brain of the crawler: the queue of URLs waiting to be fetched. Its ordering decides what gets crawled, how soon, and how often.
- Fetcher Workersworker
- Pull URLs from the frontier and download the HTML over HTTP. A large, mostly-waiting-on-the-network fleet — the crawler’s muscle.
- Parserworker
- Reads each downloaded page, pulls out the links, normalizes them, and feeds the new ones back into the frontier — that loop is the crawl.
Step 2 · Don’t crawl forever
Have I seen this URL?
The web is a cyclic graph — pages link back and forth endlessly. Without memory, the crawler re-queues the same URLs over and over and never makes progress, drowning in duplicates.
The web is cyclic — pages link back and forth. How do you avoid re-crawling forever, at billions of URLs?
An exact set of billions of URLs is hundreds of GB — too big for memory on every frontier node. You need a far more compact "have I seen this?" structure.
A bloom filter answers "probably seen / definitely not" in O(1) using a few bits per URL. It may rarely skip a new URL but never re-crawls a seen one — "mostly right and tiny" beats "exact and enormous".
Without an up-front seen-check the crawler loops on cycles and never makes forward progress — it drowns in duplicates before any dedup stage runs. Termination has to happen at enqueue time.
Before enqueuing a link, check a Seen Set. At billions of URLs, an exact set is too big for memory, so use a bloom filter: O(1) “probably seen / definitely not seen” in a fraction of the space, backed by a durable set for certainty.
- billionsURLs to track
- O(1)seen check
- ~bitsper URL
What the new pieces do
- Seen Setstore
- Remembers which URLs have already been queued so the crawler doesn’t loop forever. A bloom filter answers “seen this?” in O(1) using tiny memory.
Back of the envelope
- billions of URLs, exact set ⇒ 100s of GB
- too big to hold in memory per node
- bloom filter ⇒ ~a few bits / URL
- O(1) check in a fraction of the space
- rare false positive
- may skip a new URL, never re-crawls a seen one
Step 3 · Be a good citizen
Politeness & robots.txt
Thousands of fetchers all hammering one small site at once is indistinguishable from an attack — you’ll knock it over and get your crawler banned. And many sites declare what you may and may not crawl.
Thousands of fetchers could hammer one small site at once. How do you not look like an attack?
A global cap still lets all that traffic pile onto one unlucky host while big sites are under-crawled. Politeness is a per-domain promise, not a global throttle.
Maximum speed per host is exactly the DDoS that knocks small servers over and gets your IPs banned — ending your crawl. Speed must be bounded per host.
Obey each site’s rules and pace requests per domain, while crawling thousands of hosts simultaneously. Gentle on each, fast in aggregate — and the reason hosts become the partition unit later.
Add a Politeness layer: honor each host’s robots.txt, and enforce a per-host rate limit with a delay between requests to the same domain. Group the frontier by host so one site’s crawl is paced independently of others.
- robots.txtobeyed
- 1/hostrate limited
- parallelacross hosts
What the new pieces do
- Politenessguard
- Obeys each site’s robots.txt and caps requests per host, so the crawler is a good citizen and doesn’t accidentally DDoS a small server.
Step 4 · The hidden bottleneck
DNS at scale
Every fetch needs a hostname resolved to an IP. DNS lookups are slow (tens of ms) and, at billions of requests, become the dominant cost — and can hammer DNS servers as hard as you hammer websites.
Every fetch resolves a hostname to an IP, and DNS is slow (tens of ms). At billions of fetches, what breaks?
At billions of requests, tens-of-ms lookups become the dominant cost and can hammer DNS servers as hard as you hammer sites. Ignoring DNS makes it the silent bottleneck.
Most pages on a site share one IP, so the first lookup serves thousands of fetches; async resolution keeps fetchers from blocking. The unglamorous infra often decides throughput more than scheduling.
IPs change and rotate (CDNs, load balancers), so hard-coding breaks constantly and silently fetches the wrong server. You need caching with a TTL, not static mappings.
Put a DNS Cache in front of resolution and resolve asynchronously so fetchers don’t block. Cache by host with a TTL; most pages on a site share one IP, so the first lookup serves thousands of fetches.
What the new pieces do
- DNS Cacheservice
- Resolving hostnames is slow and would dominate latency at billions of fetches. A cache (and async resolver) keeps lookups from becoming the bottleneck.
Back of the envelope
- lookup ≈ tens of ms
- dominant per-fetch cost at billions of fetches
- 1 host ⇒ ~1 IP
- first lookup serves thousands of fetches
- cache + async resolve
- keeps the fetch pipeline full, fetchers unblocked
Step 5 · Keep what you fetched
Store & dedup content
Downloaded pages must be stored for downstream indexing — but the web is full of duplicate content: mirrors, print versions, session-ID URLs. Different URLs, identical pages. Storing them all wastes enormous space and indexing effort.
Mirrors, print versions and session-ID URLs deliver identical pages under different URLs. How do you not store the web twice?
The bloom filter dedups URLs, not content — a brand-new URL can still return a page you’ve stored a thousand times. URL-dedup and content-dedup are different jobs.
Comparing against the entire corpus per page is impossibly expensive. You need a compact fingerprint to check, not a full-content comparison.
A content hash (or near-dup signature like SimHash) checked against seen Content Hashes catches mirrors and boilerplate that are different URLs but the same page — saving storage and indexing effort.
Write raw pages to a Content Store (blob storage). Compute a content fingerprint (a hash, or a near-dup signature like SimHash) and check it against seen Content Hashes — skip pages whose content you already have under another URL.
What the new pieces do
- Content Storestore
- Durable storage (blob store) for the fetched pages — the actual payload downstream indexers and analyzers consume.
- Content Hashesstore
- Fingerprints of page content. Catches mirror sites and boilerplate that are different URLs but the same page, so you don’t store the web twice.
Step 6 · Not all pages are equal
Prioritize the frontier
Pure BFS treats a spam page like the homepage of a major news site, and never decides when to re-crawl a page that changes hourly versus one that never changes. Crawl budget is finite; spending it evenly is wrong.
Make the frontier a priority queue: rank URLs by importance (PageRank-ish signals, depth) and schedule re-crawls by how often a page actually changes. Important, fast-changing pages get crawled sooner and more often.
Step 7 · Scale & traps
Shard out, avoid traps
One machine can’t hold the frontier or fetch the web. And the web is hostile: crawler traps (infinite calendars, endless query-param permutations) can trap a naive crawler forever in one site.
One machine can’t hold the frontier or fetch the whole web. How do you shard it while keeping politeness?
Random sharding scatters one host’s URLs across many nodes, so no single node can enforce that host’s rate limit — politeness breaks. The shard key has to keep a host together.
Hashing the host to a shard puts all of a site’s URLs — and its rate limit — on one node, so per-host politeness stays local and correct. Add nodes to crawl more hosts in parallel.
Full replication means every node sees every URL and they’d duplicate work or need constant coordination to divide it. You want to partition the frontier, not copy it everywhere.
Shard the frontier by host across many machines so each shard owns some domains (keeping politeness local) and the fleet scales horizontally. Defend against traps with depth limits, URL-length caps, parameter normalization and per-host page budgets.
What the new pieces do
- Sharded Frontierbus
- The frontier spread across many machines, partitioned by host so per-host politeness is local and the whole thing scales horizontally.
Back of the envelope
- partition by host (hash)
- a site’s URLs + rate limit live on one node
- add nodes ⇒ more hosts in parallel
- horizontal scale, politeness preserved
- depth / param / length caps
- defuse infinite calendars and query-param traps
You did it
You just designed a web crawler.
Everything you assembled, in order
- A URL frontier feeding fetch → parse → enqueue: the core crawl loop.
- A bloom-filter Seen Set stops infinite re-crawling of the cyclic web graph.
- Politeness: obey robots.txt and rate-limit per host so you’re not an attack.
- A DNS cache + async resolution removes the hidden per-fetch bottleneck.
- A content store plus content-hash dedup avoids storing mirrors and boilerplate twice.
- A priority frontier balances importance and re-crawl freshness, not blind FIFO.
- Shard the frontier by host to scale out, with depth/param limits to dodge traps.