Design Web Crawler

Assume the index holds 6 billion pages and we want each one refreshed on average once every 28 days, with popular pages far more often and dead ones far less. The average HTML response is 120 KB before compression. Every page yields around 60 outlinks, most of which we have already seen.

A crawler has few external users, so the API is mostly internal: operators control jobs and scope, and fetchers lease work from the frontier. Fetcher calls are batched, because a per-URL RPC at 2,500 per second would waste more time on overhead than on downloading.

Conditional headers matter here. Sending the stored ETag or Last-Modified lets unchanged pages answer with a 304 and no body, which on a mature crawl can cut bandwidth by a third or more.

The design is a cycle, and every component sits on that cycle exactly once. The frontier decides what is fetched next. Fetchers download and nothing else. Parsers turn bytes into text and links. A dedup layer stops known URLs from re-entering the frontier and stops copied content from reaching the index. A scheduler feeds known pages back into the frontier when they are due again.

Fetching and parsing are split by a Kafka topic because they fail and scale differently. Fetchers spend their lives waiting on the network and need lots of sockets and little CPU. Parsers are the opposite. With a log between them, a parser deploy or a burst of very large pages builds up lag in Kafka instead of stalling downloads, and you can replay a day of fetched pages through a fixed parser without touching the web again.

Raw bodies go to object storage keyed by URL hash and fetch time. The Kafka message carries only the pointer and response headers, which keeps the topic small: around 2,500 messages per second of 1 KB each instead of 300 MB per second of HTML.

This is the structure the Mercator paper made popular, and it is still the cleanest answer. The front half handles importance. Each incoming URL gets a score from its inbound link count, how stale our copy is, how often the page has changed in the past and which job asked for it. A weighted random pick across a few priority bands means background URLs still drain slowly instead of starving.

The back half handles politeness. Every host maps to exactly one back queue on one fetcher node. A min-heap keyed by the earliest time each host may be contacted again tells a fetcher thread which queue to service. After a fetch, the host's next time is set to now plus the larger of its robots crawl-delay and a multiple of how long the last response took, so slow servers automatically get gentler treatment.

Keep the back queues short, a few hundred URLs per host, and refill them from the front when they drain. A long back queue for one giant host is wasted memory, because that host can only take one request every second or two anyway.

Forty billion URLs do not fit in memory, but the part that matters right now does. Store the full URL table in a partitioned on-disk store such as RocksDB or Cassandra, keyed by host then URL hash. Each frontier partition keeps only the next few hours of due work in memory and pages more in as queues drain. Partition by host hash, the same key the fetchers use, so every component that touches a host agrees on where that host lives.

When a fetcher node dies, its host buckets move to other nodes through consistent hashing. The new owner reads the hosts' last-fetch times from the URL store before sending anything, so a reassignment never causes a burst against a site that was just contacted.

Layer 2 handles the 150,000 links a second. A Bloom filter never says "new" for a URL it has seen, so it never causes a duplicate fetch. It sometimes says "seen" for a URL it has not, which means we occasionally skip a genuinely new page. At a 1% rate that is acceptable, because important pages are linked from many places and will be offered again. For URLs from sitemaps or high-value seeds, bypass the filter and check the URL store directly.

Layer 4 is what protects the index from mirrors, syndicated news and printer-friendly copies. SimHash turns a document's weighted word features into 64 bits where similar documents differ in only a few bit positions. Split the fingerprint into four 16-bit bands. Any two documents within three bits of each other must match exactly on at least one band, so you only compare against documents that share a band. When a near copy is found, keep the version with the stronger link signal as canonical and record the other as duplicate_of.

A fixed revisit interval wastes most of the budget. Track, per URL, how many of the last N fetches found changed content, and estimate a change rate. Schedule the next visit so the expected staleness is roughly equal across pages of similar importance. A page that changed on 9 of the last 10 visits halves its interval. One that has not changed in six visits doubles it, up to a cap. A 304 response counts as "unchanged" at almost no cost, so conditional requests make this feedback cheap to collect.