Design Google News

Assume 150 million daily readers who open the app six times a day, 60,000 publishers, and five years of articles kept searchable.

Read the diagram as two halves joined by the story store. The left half is a pipeline that turns raw web pages into scored stories; it runs continuously and can lag by a minute without anyone noticing. The right half is the serving path; it must answer in milliseconds and keep working even if the left half stalls.

The crawl scheduler knows every publisher's feeds and how often each one tends to publish, and polls busy feeds every minute and quiet ones every half hour. Publishers that support WebSub push new items to us instead. Fetchers download the article pages, honouring per-host limits. Extraction strips navigation and ads, finds the headline, author, publish time and canonical URL, detects the language, and drops exact and near duplicates. Enrichment tags named entities, assigns topics and computes a text embedding.

The clusterer assigns each new article to an existing story or starts a new one. The scorer recomputes each active story's importance as coverage grows. Once a minute, the edition builder takes the top scored stories for every region, language and section, applies diversity rules, and writes a ready-made edition to a cache. The feed service either returns that edition as is, or mixes it with personal candidates and re-ranks for a signed-in reader.

A large share of articles are near copies: wire stories republished with a new headline, or the same article reachable at three URLs. Handle URL variants with canonical tags and parameter stripping. For body text, compute a 64-bit SimHash over word shingles; two articles whose hashes differ in three bits or fewer are treated as the same text. Duplicates are not discarded. They are attached to the original as extra evidence that the story is spreading, which matters for scoring, but they never appear on the page as separate entries.

Each active story keeps a centroid embedding, a set of weighted entities (people, organisations, places) and a time range. When an article arrives, the clusterer finds candidate stories with an approximate nearest neighbour search over centroids from the last 72 hours, limited to the same language or with cross-lingual embeddings if you support that. It then scores each candidate on embedding similarity, entity overlap and time gap. Above a threshold, the article joins the best story and the centroid updates as a running average. Below it, the article starts a new story.

Entities act as a guard rail. Two articles about different football matches on the same night have similar embeddings, but different team names. Two articles about the same earthquake in different languages look different lexically but share a place and a magnitude. Requiring some entity overlap reduces both false merges and missed matches.

Online assignment is greedy, so it makes mistakes that compound. Two clusters form for one event because the first articles used different terms; a cluster drifts as later articles shift focus and starts absorbing unrelated pieces. A repair job runs every 15 minutes over active stories and nightly over the full window, using a more careful algorithm such as agglomerative clustering on the full article set. It merges stories whose centroids converged, splits stories whose internal similarity dropped, and retires stories with no new coverage for two days. Story ids are kept stable through merges by redirecting the absorbed id to the survivor.

A story's score blends four signals. Breadth: how many distinct, reputable publishers cover it, counted after duplicates collapse. Velocity: how quickly that count is growing over the last hour, which is what makes a breaking story jump the queue. Source quality: a per-publisher authority score maintained from editorial standards, corrections history and expert review, not from clicks. Age: an exponential decay with a half-life of a few hours for news and longer for features.

One way to write it: score = (log(1 + quality_weighted_outlets) + 2 x velocity) x 0.5^(age_hours / 6). The exact formula is less important than its properties: a story covered by forty serious outlets beats one covered by four hundred copies of a press release, and a story that stops growing sinks within a day.

Pure score order produces a front page with six stories about the same election and the same outlet leading four of them. The edition builder re-ranks greedily: at each slot, pick the story that maximises score minus a penalty for similarity to stories already chosen, a version of maximal marginal relevance. Add hard rules on top: at most two lead articles from one publisher per screen, at least one story from each major section in the first twenty, and lead article rotation within a story so the same outlet does not always win.

Within a story, the lead article is chosen for quality and originality, preferring the outlet that broke the news or has first-hand reporting. Coverage beneath it deliberately includes analysis, local reporting and outlets with different editorial leanings, which is a product choice as much as a ranking one.

For signed-in readers, the feed service starts with about 200 stories from the shared edition and adds about 100 personal candidates: stories matching followed topics and sources, and stories close to the reader's interest vector, which is an average of embeddings of recently read stories with decay. A light ranking model scores all 300 using story score, topical match, recency of the reader's interest and whether they already saw the story elsewhere. Diversity rules run again on the result. If the profile or ranking service is slow, the feed falls back to the edition, which is always available.