Design Google News

Four pipelines isolated by concern:

• Crawl tier: Distributed crawlers fetch articles from 50K+ sources on adaptive schedules. URL frontier prioritizes sources by publish frequency. Respects robots.txt; backs off on errors.
• Content tier: Kafka-backed pipeline extracts clean text, detects language, classifies topic, extracts named entities, and computes SimHash fingerprint for cluster matching.
• Ranking tier: Scores clusters by authority x freshness decay, personalizes by user interest vector (128-dim topic embedding from click history), and applies MMR diversity constraint.
• Serving tier: Pre-computed feeds cached in Redis with 5-min TTL. On-request lightweight re-rank injects breaking news. CDN caches trending and topic pages.

Offline ML trainer closes the loop: click/dwell data feeds back into the ranking model weights and user interest vectors via nightly batch jobs.

(a) Near-duplicate clustering. When 200 outlets publish "President signs climate bill," the user should see one card with "200+ sources." We use SimHash (or MinHash) on article text to produce a 64-bit fingerprint. Two articles with Hamming distance ≤ 3 are considered near-duplicates.

-- SimHash clustering pseudocode
fingerprint = simhash(article.text)          -- 64-bit hash
candidates  = lookup_lsh(fingerprint, k=3)   -- LSH index: find hashes within Hamming distance 3
if candidates:
    best_cluster = closest(candidates)
    merge(article, best_cluster)              -- add to existing story cluster
else:
    create_cluster(article)                   -- new story
-- representative = highest-authority article in cluster

Each cluster stores: representative article (highest authority score), source list, earliest publish time, and topic labels. The feed shows the representative with "N sources" badge.

Why SimHash over MinHash? SimHash produces a single 64-bit fingerprint per document -- constant space regardless of document length. MinHash produces a signature of k hashes (typically k=128), more accurate for Jaccard similarity but 128x more storage. For news articles (similar length, mostly text), SimHash at distance 3 achieves ~95% recall with far less storage. We use MinHash as a secondary offline check for borderline cases (Hamming distance 3-5).

LSH index structure. We split the 64-bit SimHash into 4 bands of 16 bits. Two documents that share at least one identical 16-bit band are candidates. This gives sub-millisecond lookups against the 50M active cluster fingerprints, stored in memory across a sharded hash table.

(b) Freshness decay. A breaking-news article from 10 minutes ago should rank far above yesterday's story. We model this with exponential decay:

score = authority_score * e^(-lambda * age_hours)

-- Breaking news:  lambda = 0.5  (half-life ~1.4 hours)
-- Regular news:   lambda = 0.1  (half-life ~7 hours)
-- Evergreen:      lambda = 0.02 (half-life ~35 hours)

The system classifies each article's decay rate based on topic and velocity (how fast new articles join the cluster). A cluster gaining 50 new articles/hour gets breaking-news lambda.

Adaptive lambda selection. A simple heuristic: if cluster.article_count_last_hour > 20, use breaking lambda (0.5). If the article's topic is in ["obituary", "historical", "explainer"], use evergreen lambda (0.02). Default: regular (0.1). The ML trainer can also learn per-topic lambda from engagement data -- topics where users prefer recency get higher lambda.

(c) Diversity via greedy MMR. Without diversity constraints, a user interested in politics would see 20 politics articles from CNN. We apply Maximal Marginal Relevance:

selected = []
for i in range(top_K):
    best = argmax over candidates:
        alpha * relevance(candidate, user)
        - (1 - alpha) * max_similarity(candidate, selected)
    selected.append(best)
-- Constraint: no more than 2 articles from same source in top-20

Each pick maximizes relevance to the user while minimizing similarity to already-selected articles. Alpha = 0.7 balances relevance vs diversity. Hard cap: max 2 articles per source domain in the top 20.

MMR in practice. The candidate pool is ~500 top-scoring clusters after the initial ranker pass. MMR re-ranks these into the final top-20. Similarity is computed as cosine distance between cluster topic-embedding vectors (pre-computed, 128-dim). The full MMR loop runs in < 5 ms for 500 candidates -- fast enough for on-request computation. This is the key to preventing "5 articles about the same politician" feeds.

Source diversity enforcement. Beyond MMR's soft diversity, we apply a hard constraint: after selecting 2 articles from source X, all remaining candidates from source X are removed from the pool. This guarantees no single outlet dominates the feed, even if the user clicks CNN articles exclusively.

flowchart LR
    A[Article Crawled] --> B[Content Pipeline]
    B --> B1[Text Extract + Clean]
    B1 --> B2[Entity + Topic Classify]
    B2 --> C[SimHash Fingerprint]
    C --> D{Cluster MatchHamming dist lte 3?}
    D -->|Yes| E[Merge into Existing Cluster]
    D -->|No| F[Create New Cluster]
    E --> G[Update Cluster Metadata]
    F --> G
    G --> H[Ranker: authority x freshness decay]
    H --> I[Top-K per User with MMR Diversity]
    I --> J[Feed Cache - Redis 5min TTL]
          

• Pre-compute personalized feed vs compute on request. Pre-compute: fast serving (~50 ms from Redis), but stale by up to 5 min. On-request: always fresh, but 200-500 ms latency and higher compute cost. Hybrid: pre-compute base feed, apply lightweight re-rank on request for breaking news injection.
• SimHash (fast, approximate) vs exact dedup (slow, precise). SimHash at Hamming distance 3 catches ~95% of near-duplicates with < 1 ms per lookup. Exact comparison (TF-IDF cosine) catches 99% but costs ~50 ms. SimHash for real-time pipeline; exact dedup as offline cleanup job.
• Aggressive personalization vs diverse feed. Deep personalization creates filter bubbles -- user only sees topics they already like. Diverse feed includes serendipitous discovery but may feel less relevant. Tunable alpha in MMR: 0.9 = heavy personalization, 0.5 = balanced exploration.
• Crawl depth vs latency. Deep-crawling (following links within articles) yields richer content and related articles but adds minutes to ingestion latency. Shallow crawl (RSS + landing page only) is faster but misses context.
• Real-time trending vs batch trending. Real-time (sliding window on Kafka stream) detects trends in minutes but is compute-heavy. Batch (hourly MapReduce) is cheaper but misses fast-moving stories. Use real-time for top-of-feed "breaking" slot; batch for topic pages.
• Source-level authority vs article-level quality. Source authority (domain reputation) is stable and cheap to compute but penalizes good articles from low-authority sources. Article-level quality (readability, factual density, expert quotes) is more accurate but requires NLP inference per article. Blend: 70% source authority + 30% article quality signal.
• Single global ranker vs per-region rankers. Global ranker simplifies operations but cannot capture regional editorial norms (e.g., tabloid-style is normal in UK, not in Japan). Per-region rankers allow tuning but multiply model-training cost. Compromise: global base model + region-specific feature weights.

1. Lead with clustering. "The key insight is that 200 outlets publish the same story -- we cluster near-duplicates via SimHash and show one representative per cluster." This immediately shows you understand what makes news aggregation different from generic feed.
2. Name the freshness model. "score = authority x e^(-lambda x age) where lambda varies by article type." Concrete formula beats hand-waving about "freshness matters."
3. Explain MMR for diversity. "Greedy Maximal Marginal Relevance -- each pick maximizes relevance while minimizing similarity to already-selected items." Shows you know recommender-system theory.
4. Adaptive crawl rate is the politeness story. Don't just say "we crawl." Say "CNN every 2 min, local blog every 6 hours, based on learned publish frequency." Shows operational maturity.
5. Distinguish from social feed. News feed ranks professional content by authority + freshness. Social feed ranks user-generated content by engagement + social graph. Different ranking signals, different abuse vectors.
6. Mention the cold-start problem. New user with no click history: fall back to location-based trending + globally popular stories. After ~20 clicks, the interest vector has enough signal for personalization. Explicit topic follows accelerate cold-start.
7. Crawl-to-display latency is a differentiator. "Breaking news visible in < 5 min: priority re-crawl triggered by trending velocity spike, plus push-based ingestion from wire services (AP, Reuters)." Shows you think about the full pipeline end-to-end.