Social & Feed
Design Twitter Trending
01
Requirements
Requirements
- Identify trending terms — hashtags, bigrams, named entities — from all tweets
- Ranking by "trending-ness": how much more frequent NOW than baseline
- Per-geo trends (country, city), per-language
- Personalized trends based on user's follow graph + interests
- Top-10 list refreshed every ~5 min per scope
- Attach a tweet / summary preview to each trend (what's the story)
- End-to-end latency from tweet → trends recompute: < 1 min
- Scale to ~6K tweets/sec sustained; peaks 20K+ for major events
- 400+ geo scopes × ~10 languages × personalization → millions of trend lists
- Resilient to coordinated manipulation (bot rings, hashtag hijacking)
- Memory-efficient: can't keep exact counts for every term (billions of unique terms)
- Eventually consistent; approximations are fine if close to truth
02
Scale Estimation
Scale Estimation
03
API Design
API Design
Get trends for a Where-On-Earth ID (geo code). Returns {as_of, trends: [{name, volume, preview_tweet_id, url}], locations: [{country, ...}]}. Served from pre-computed cache; < 50 ms response.
Returns trends ranked by what's relevant to the caller's graph + interests. Requires auth; combines global trending with follow-graph signal.
When user taps a trend: search for that term. Top tweets attached to the trend list come from this search run at trend-refresh time.
Manually block a term from trending (policy violations, hate speech, spam). Small escape hatch layered on top of ML-based filtering.
List all WOEID locations that have trend data. Used by clients to populate the geo picker.
04
Architecture
Architecture
A streaming pipeline. Tweets flow through Kafka; a feature extractor emits term events; a sketch service maintains approximate counts per (term, geo, window); a trend ranker runs every few minutes to compute top-K per geo, filter spam, pick a preview tweet, and write to the serving cache. Clients read from the cache only — no expensive queries on the hot path.
05
Deep Dive — Top-K on a Firehose + "Trending" as Deviation
Naive top-K: keep a dict term→count; at refresh time, sort and take top 10. Breaks immediately — at 90K term-events/sec, the dict is 50M+ unique keys/day; sorting 50M entries per geo per minute is untenable.
Count-Min Sketch + heavy-hitter tracker. A Count-Min Sketch is a probabilistic counter: constant memory, approximate counts with small over-estimation bias. Paired with a bounded min-heap of top-K heavy hitters, it gives us top-K in fixed memory.
"Trending" means deviation from baseline, not raw volume. #love is frequent every day — it's not news. A new spike of a previously-rare term IS news. Rank by:
This EMA (exponentially-weighted moving average) is the baseline — cached per (term, geo) for frequent terms, defaulted for never-before-seen ones. The ratio metric surfaces true spikes, not the ambient background.
flowchart LR T[Tweet] --> X[Extractor tokens+entities] X --> S[Spam filter bot score] S -->|pass| C[CMS counter 5-min window] C --> H[Heavy-hitter min-heap] H --> R[Ranker current/baseline] B[Baseline EMA past 7 days] --> R R --> D[Dedup & filter policy block] D --> P[Preview picker top tweet / LLM] P --> K[Trend cache Redis per geo]
Spam / bot filtering. Bots can trivially game raw counts. Each user has a user reputation score (follower ratio, age, engagement authenticity signals); a tweet's contribution is weighted. Additionally, term events are deduped by near-hash (prevent copy-paste spam) and clustered by author — if 100 tweets come from 10 accounts with follow-overlap = 1.0, that's a bot ring; down-weight drastically.
Dedup + entity resolution. "#BarackObama", "#Obama", "Barack Obama" should merge. Entity resolution via a lightweight NLP pipeline (knowledge-graph lookup + fuzzy matching) clusters surface forms into canonical trends. Otherwise the top-10 is noisy and overlapping.
Preview pick. Once ranker decides a term is trending, it runs a quick search for recent high-engagement tweets matching that term; picks one as the preview. Modern systems (2024+) use an LLM to write a 1-line summary ("Markets tumbled after the Fed's rate announcement") instead of just picking a tweet.
06
Tradeoffs & Design Choices
Tradeoffs & Design Choices
- Approximate (CMS) vs exact counts. Exact counts require 10–100× more memory for no meaningful accuracy gain in top-K. CMS's 1–5% over-estimation is invisible at top-K granularity.
- Raw frequency vs deviation ranking. Raw frequency = "#love always wins." Deviation (current / baseline) is what makes trends feel like news. But deviation is noisy for low-count terms; floor the rate to avoid "brand new term with 10 tweets beats viral trend with 100K" false positives.
- 5-min windows vs 1-min or 1-hour. 5 min is the UX sweet spot: responsive but stable. 1 min causes flicker; 1 hour misses breaking news. Keep multiple window sizes if needed for different products.
- Geo detection. IP-geo for anonymous users; profile location for logged-in. Both are unreliable. Mitigation: use multiple signals + cluster by language; don't show "#MelbourneCupDay" globally.
- Centralized ranker vs distributed. Ranking every geo from one process is simpler; distributing by geo shards is necessary at planet scale. Each geo computes its own top-K; a thin aggregator handles global/cross-geo "world" trends.
- Human-in-the-loop. ML alone isn't enough. Teams maintain deny-lists for slurs, false positives, obvious manipulation. Be explicit about this in interviews — pretending ML solves content safety is naive.
07
Failure Modes
Failure Modes
08
Evolution
Evolution
MVP — exact counts in Redis
Hashmap of term → count with per-minute buckets. Sort + take top 10. Works to ~10M tweets/day.
Count-Min Sketch + top-K heap
Switch to probabilistic counts. Handle ~100M+ unique terms/day in fixed memory. Per-geo sketches.
Deviation-based ranking + EMA baseline
"Trending" = current / baseline. No more "#love tops every day." 7-day EMA cached per (term, geo).
Spam filtering + entity resolution
User-reputation-weighted counts. Bot-ring detection via graph clustering. Entity merging for surface-form variants.
Personalized + LLM summaries
Trends reranked per user based on follow graph and interests. LLM generates 1-line description per trend ("Markets tumbled after Fed rate cut"). Modern X Explore tab.
Watch and read
References & Videos
Try next
Free to read · better with Enzo
Whiteboard this with Enzo
Enzo runs it as a live system design round on the whiteboard and grades your trade-offs.