The big idea
What is a web crawler?
A crawler downloads the web — start from a few URLs, fetch each page, find its links, fetch those too, forever. Trivial as a weekend script; brutal at billions of pages, where you must be fast, polite, dedup-aware, and never get stuck in a loop or a trap.
Model it as one big loop over a queue: pull a URL, fetch it, extract links, enqueue the new ones. Then make that loop scale, behave, and stay efficient. Everything we add defends that simple loop against the messy reality of the web.
How to read this: We add one piece at a time, problem then fix, and the diagram grows. Hit Begin.
Step 1 · The loop
Fetch, parse, enqueue
At its heart a crawl is a graph traversal: you have URLs to visit, you visit one, it reveals more URLs, repeat. You need somewhere to hold the “to-visit” list and workers to do the visiting.
Design decision: A crawl reveals more URLs as it fetches. How do you structure the loop?
The call: A URL Frontier queue; fetchers and a parser loop over it. — The frontier holds what to crawl next; fetchers download, the parser extracts links and pushes new ones back. Decoupling fetch (network-bound) from parse (CPU-bound) lets each scale on its own.
A URL Frontier holds what to crawl next. Fetcher workers pull from it and download pages; the Parser extracts links and pushes the new ones back into the frontier. Seeds prime the loop and it runs itself from there.
BFS over the web graph: The frontier is just a queue, so the default traversal is breadth-first from the seeds. Decoupling fetch from parse lets each scale independently — fetching is network-bound, parsing is CPU-bound.
Step 2 · Don’t crawl forever
Have I seen this URL?
The web is a cyclic graph — pages link back and forth endlessly. Without memory, the crawler re-queues the same URLs over and over and never makes progress, drowning in duplicates.
Design decision: The web is cyclic — pages link back and forth. How do you avoid re-crawling forever, at billions of URLs?
The call: A bloom filter as the Seen Set (backed by a durable set). — A bloom filter answers "probably seen / definitely not" in O(1) using a few bits per URL. It may rarely skip a new URL but never re-crawls a seen one — "mostly right and tiny" beats "exact and enormous".
Before enqueuing a link, check a Seen Set. At billions of URLs, an exact set is too big for memory, so use a bloom filter: O(1) “probably seen / definitely not seen” in a fraction of the space, backed by a durable set for certainty.
Bloom filters: A bloom filter trades a tiny false-positive rate for huge memory savings — it may occasionally skip a new URL but never re-crawls a seen one. Perfect when “mostly right and tiny” beats “exact and enormous”.
Step 3 · Be a good citizen
Politeness & robots.txt
Thousands of fetchers all hammering one small site at once is indistinguishable from an attack — you’ll knock it over and get your crawler banned. And many sites declare what you may and may not crawl.
Design decision: Thousands of fetchers could hammer one small site at once. How do you not look like an attack?
The call: Honor robots.txt + per-host rate limit, crawl many hosts in parallel. — Obey each site’s rules and pace requests per domain, while crawling thousands of hosts simultaneously. Gentle on each, fast in aggregate — and the reason hosts become the partition unit later.
Add a Politeness layer: honor each host’s robots.txt, and enforce a per-host rate limit with a delay between requests to the same domain. Group the frontier by host so one site’s crawl is paced independently of others.
Per-host, not global: Politeness is a per-domain promise. The frontier must let you crawl many hosts in parallel while staying gentle on each — which is exactly why hosts become the natural unit of partitioning later.
Step 4 · The hidden bottleneck
DNS at scale
Every fetch needs a hostname resolved to an IP. DNS lookups are slow (tens of ms) and, at billions of requests, become the dominant cost — and can hammer DNS servers as hard as you hammer websites.
Design decision: Every fetch resolves a hostname to an IP, and DNS is slow (tens of ms). At billions of fetches, what breaks?
The call: Cache resolutions per host (TTL) and resolve asynchronously. — Most pages on a site share one IP, so the first lookup serves thousands of fetches; async resolution keeps fetchers from blocking. The unglamorous infra often decides throughput more than scheduling.
Put a DNS Cache in front of resolution and resolve asynchronously so fetchers don’t block. Cache by host with a TTL; most pages on a site share one IP, so the first lookup serves thousands of fetches.
Cache the boring stuff: The unglamorous infrastructure — DNS, connection reuse — often decides crawler throughput more than clever scheduling. Resolve once per host, reuse connections, keep the network pipeline full.
Step 5 · Keep what you fetched
Store & dedup content
Downloaded pages must be stored for downstream indexing — but the web is full of duplicate content: mirrors, print versions, session-ID URLs. Different URLs, identical pages. Storing them all wastes enormous space and indexing effort.
Design decision: Mirrors, print versions and session-ID URLs deliver identical pages under different URLs. How do you not store the web twice?
The call: Fingerprint content (hash / SimHash) and skip pages you already have. — A content hash (or near-dup signature like SimHash) checked against seen Content Hashes catches mirrors and boilerplate that are different URLs but the same page — saving storage and indexing effort.
Write raw pages to a Content Store (blob storage). Compute a content fingerprint (a hash, or a near-dup signature like SimHash) and check it against seen Content Hashes — skip pages whose content you already have under another URL.
URL-dedup ≠ content-dedup: The bloom filter dedups URLs; this dedups content. You need both: a fresh URL can still deliver a page you’ve already stored a thousand times.
Step 6 · Not all pages are equal
Prioritize the frontier
Pure BFS treats a spam page like the homepage of a major news site, and never decides when to re-crawl a page that changes hourly versus one that never changes. Crawl budget is finite; spending it evenly is wrong.
Make the frontier a priority queue: rank URLs by importance (PageRank-ish signals, depth) and schedule re-crawls by how often a page actually changes. Important, fast-changing pages get crawled sooner and more often.
Freshness vs coverage: A crawler constantly trades discovering new pages against refreshing known ones. Encoding both into frontier priority — not a fixed FIFO — is what separates a toy from a search-engine crawler.
Step 7 · Scale & traps
Shard out, avoid traps
One machine can’t hold the frontier or fetch the web. And the web is hostile: crawler traps (infinite calendars, endless query-param permutations) can trap a naive crawler forever in one site.
Design decision: One machine can’t hold the frontier or fetch the whole web. How do you shard it while keeping politeness?
The call: Shard the frontier by host across machines. — Hashing the host to a shard puts all of a site’s URLs — and its rate limit — on one node, so per-host politeness stays local and correct. Add nodes to crawl more hosts in parallel.
Shard the frontier by host across many machines so each shard owns some domains (keeping politeness local) and the fleet scales horizontally. Defend against traps with depth limits, URL-length caps, parameter normalization and per-host page budgets.
Partition by host: Hashing the host to a shard means all of one site’s URLs — and its rate limit — live on one node. Add nodes to crawl more hosts in parallel; the per-host promise still holds.
You did it
You just designed a web crawler.
- A URL frontier feeding fetch → parse → enqueue: the core crawl loop.
- A bloom-filter Seen Set stops infinite re-crawling of the cyclic web graph.
- Politeness: obey robots.txt and rate-limit per host so you’re not an attack.
- A DNS cache + async resolution removes the hidden per-fetch bottleneck.
- A content store plus content-hash dedup avoids storing mirrors and boilerplate twice.
- A priority frontier balances importance and re-crawl freshness, not blind FIFO.
- Shard the frontier by host to scale out, with depth/param limits to dodge traps.