Social & Feed
Design Twitter Trending
01
What the trends service must do
What the trends service must do
- Pull candidate terms out of every public post: hashtags, short phrases and named entities, normalised so that spelling and casing variants merge.
- Rank terms by how far their current rate sits above what is normal for that term at this hour and in this place, not by raw count.
- Publish separate lists for the whole world, each country, roughly 600 metro areas and each major language.
- Offer a "For you" list that leans on the viewer's regions and interests, built from the same candidate pool.
- Each trend card shows a display label, a category such as Sports or Politics, a rounded post volume and a few representative posts.
- Trust and safety staff can suppress a term everywhere within seconds, and sensitive candidates go to a review queue before they appear.
- A genuine surge should reach the panel within 3 minutes at p95 from the moment it starts.
- Reading a trend list stays under 40 ms at p99, served from precomputed results.
- Counting memory is bounded and does not grow with the number of distinct terms people type.
- Small coordinated groups must not be able to buy a slot: the signal is people, not posts.
- If the pipeline stalls, readers see the last good list with its age, never an empty panel and never a list hours old without a warning.
Pin down how fresh "now" must be, how many scopes exist, whether personal lists are required in the first version, and how much moderation sits between the ranker and the screen. A 3-minute target forces streaming; a 30-minute target lets you run batch jobs and skip half of this design. Ask whether post counts shown on cards must be exact. They almost never are, and saying so early lets you use approximate counting without apology.
02
Sizing the firehose
Sizing the firehose
Assume 720 million public posts a day and about six candidate terms per post after stop words are removed. A post counts towards the global list, its country, its metro when known, and its language, so on average 3.5 scopes. On the read side, 250 million daily users open the trends panel about six times a day.
| Quantity | Working | Result |
|---|---|---|
| Posts per day | assumption | 720M |
| Average post rate | 720M / 86,400 s | about 8,300 / s |
| Peak post rate | 4x average during a final or an election night | about 33,000 / s |
| Term events at peak | 33K posts x 6 terms | about 200,000 / s |
| Counter updates at peak | 200K term events x 3.5 scopes | about 700,000 / s |
| Distinct terms seen per day | long tail, most appear once | about 120M |
| Scopes | 1 global + 190 countries + 600 metros + 60 languages | about 850 |
| Sketch memory per minute | global 20 MB, country or language 1 MB each, metro 256 KB each | about 420 MB |
| Sketch ring for one hour | 420 MB x 60 one-minute slots | about 25 GB across the counting fleet |
| Tracked candidates | 850 scopes x 2,000 candidates, minute counts + author HLL | 1.7M entries, about 3 GB |
| Baseline table | 40M term and scope pairs x 24 hourly slots x 8 B | about 7.7 GB |
| Panel reads | 250M users x 6 loads / 86,400 s, peak 3.5x | 17,000 / s average, 60,000 / s peak |
Writes into the counters reach 700,000 updates per second, but each one is a few memory increments, so a few dozen counting workers handle it. Reads are trivial because there are only 850 lists to serve. The real constraint is memory per distinct term: 120 million terms a day multiplied by 850 scopes and 60 minute buckets is far beyond any exact table, which is why the design leans on sketches.
The whole output of the system is under 10 MB: 850 lists of 50 entries, each a few hundred bytes. It fits in every Redis replica and every CDN node.
03
API design
API design
Clients never ask the system to compute anything. Every read is a key lookup on scope, which is why the API can sit behind a CDN with a 30-second TTL. Volumes are returned as buckets such as "25K+ posts" so that approximate counts are never shown as precise numbers.
04
Architecture overview
Architecture overview
flowchart LR
P["Post service"] --> K[["Kafka: posts"]]
K --> X["Term extractor"]
X --> G["Quality gate"]
G --> K2[["Kafka: term events by scope"]]
K2 --> C["Counting workers"]
C --> R["Trend ranker, every 60 s"]
B[("Baseline store")] --> R
C -. "minute rollups" .-> W[("Columnar history")]
W --> N["Nightly baseline job"]
N --> B
R --> F["Policy filter and clustering"]
F --> S[("Trend lists in Redis")]
S --> A["Trends API and CDN"]The pipeline is a chain of stream stages, each of which shrinks the data. Roughly 33,000 posts a second come in at peak; a few thousand candidate rows a minute reach the ranker; 850 small lists come out. Every stage reads from a Kafka topic and writes to the next one, so any stage can fall behind, be redeployed or be replayed without touching the others.
The term extractor tokenises each post, detects its language, pulls hashtags and two or three word phrases, and runs entity linking so that "Messi", "#Messi" and "Leo Messi" become one entity ID. It also resolves the post to scopes using the author's home region, an attached place, or coarse IP location as a last resort. The quality gate attaches an author trust score and drops near-duplicate text using SimHash, so a thousand copies of one message count once.
Term events are keyed by scope before counting. That puts all of Mumbai on one worker and spreads the global scope across several, which keeps sketches local and avoids cross-worker merges on the hot path. The ranker reads the current minute counts, compares them with stored baselines and writes a fresh list per scope. A slow nightly job recomputes those baselines from the columnar history, so the expensive statistics never run in the streaming path.
Stateless and horizontally scaled. It holds a stop-word list per language and an entity dictionary of about 30 million names, refreshed daily. Output is about six term events per post.
Each owns a set of scopes and keeps a ring of 60 one-minute Count-Min Sketches plus a candidate table of the top 2,000 terms with exact minute counts and a HyperLogLog of distinct authors.
Runs once a minute per scope. Scores every candidate against its baseline, applies quality multipliers and emits the top 50. A scope takes milliseconds, so one ranker process can cover hundreds.
Applies blocklists and the suppression table, routes sensitive terms to review, and merges terms that co-occur heavily into one trend with one label.
Wide-column store keyed by term and scope. Holds a smoothed mean and variance for each hour of the day, plus a weekly correction factor, so "#MondayMotivation" is expected on Mondays.
Redis holds the current and previous list per scope. The API adds personal blending and the CDN absorbs nearly all anonymous traffic.
05
Deep dive: bounded counting and the surprise score
Deep dive: bounded counting and the surprise score
How do you count hundreds of millions of distinct terms per minute per region in fixed memory? And once you have counts, what number makes a term "trending" rather than merely common? The answers are a two-tier counter and a surprise score.
flowchart TD
E["Term event arrives"] --> CM["Add to current minute sketch"]
CM --> T{"Already a candidate?"}
T -->|yes| EX["Increment exact count and author HLL"]
T -->|no| EST{"Sketch estimate over admission floor?"}
EST -->|no| DROP["Stay in sketch only"]
EST -->|yes| ADM["Admit, evict weakest candidate"]
ADM --> EX
EX --> RK["Read by ranker at minute close"]Tier one: a sketch for every term
A Count-Min Sketch is a small grid of counters, d rows by w columns. Each row has its own hash function. To add a term you increment one cell per row; to estimate it you take the smallest of those cells. Collisions can only add to a cell, so the estimate never undercounts, and the overcount is bounded by about e / w times the total number of events in the window.
For the global scope we use 5 rows by 220 columns of 4-byte counters, which is 20 MB per minute. A busy global minute holds about 3 million term events, so the expected overcount is roughly 8 events per term, invisible next to anything that could trend. Smaller scopes use narrower sketches because their totals are smaller. We use conservative update: only the cells that currently hold the minimum are raised. It costs a few reads per update and cuts the overcount on long-tail terms noticeably.
Sketches merge by adding cells, so a 5-minute view is just the sum of five minute sketches. After an hour, minute sketches are folded into one hourly sketch and dropped, which keeps the ring at 25 GB fleet-wide.
Tier two: exact counts for the few that matter
A sketch can answer "how many times did X appear", but it cannot list which terms are big. So each scope also keeps a candidate table of 2,000 entries. When an event arrives for a term that is not a candidate, we check its sketch estimate for the last five minutes; if it beats the weakest candidate, the term is admitted and the weakest is evicted. This is the Space-Saving idea with a sketch as the gate.
Candidates get exact per-minute counts and a HyperLogLog of author IDs at precision 11, about 1.5 KB each, giving distinct-author counts within about 2%. Distinct authors, not posts, are what the ranker reads.
Trending means surprising
For each candidate the baseline store returns the expected count μ for this term, this scope and this hour of the week, plus its variance. The score asks how unlikely the observed count is under that expectation. A Poisson-style score works well and handles terms with no history at all:
The floor on μ matters. Without it a brand-new hashtag used by four people scores as an infinite deviation. The acceleration factor lets an emerging term outrank one that peaked an hour ago, and the diversity factor punishes one account posting the same tag fifty times. The final ranking also applies a decay so that a trend must keep earning its slot: each minute its displayed score is blended 70/30 with the previous one, which stops the list from jittering every refresh.
One story, one card
During a cricket final you will see "#INDvAUS", "Kohli", "World Cup final" and "#CWC26" all spike together. Showing four cards for one event wastes the panel. The clustering step looks at co-occurrence inside the candidate set: if two terms share more than 40% of their recent posts, they join one cluster. The label comes from the member with the most distinct authors, preferring an entity name over a hashtag when both are close. The category comes from the entity graph or a light text classifier.
Representative posts are chosen from authors with high trust scores, with media or links that already passed safety scanning, and never from accounts younger than 30 days. A bad preview post is the most visible failure this system can have, so the picker is deliberately conservative.
Every defence above is built into the numbers rather than bolted on after: distinct authors instead of posts, a trust weight per author, SimHash to collapse copy-paste, and a cap on how much any single account cluster can add to one term. A campaign of 5,000 fresh accounts barely moves a score that a few hundred real people would move.
06
Decisions and their costs
Decisions and their costs
Memory stays fixed no matter how many strange strings people type, and the terms that might trend get precise counts and author tracking. The price is a small, one-sided error on the long tail, which never matters for ranking.
Bounded memoryA stream processor with keyed state can count everything exactly, and at small scale it is simpler. At 120 million terms a day across 850 scopes it means billions of state entries, heavy checkpoints and slow recovery after a worker restart.
Fine for one countrySteady terms stay off the list, recurring weekly tags are expected, and a new term must clearly exceed its floor. The cost is a baseline store and a nightly job to keep it fresh.
Matches what users meanTrivial to compute and easy to explain, but the list fills with greetings, sports club names and permanent fan tags. It also rewards whoever posts most, which is exactly what spammers can do.
Wrong signalA surge reaches the list in two or three cycles, inside the 3-minute target. Scoring on a 5-minute sliding view built from minute buckets keeps noise down while preserving speed.
Fresh and stableCheaper, smoother and good enough for a weekly digest. For breaking news it is too slow: an earthquake would trend after people had already found it elsewhere.
Use for digestsOnly 850 lists are computed. "For you" merges the viewer's scopes and re-orders by topic affinity at read time, which costs one small model call and no extra pipeline.
Cheap personalisationRunning surprise scoring over each user's own network sounds attractive, but it multiplies the work by the user count and produces lists that feel like a second home timeline rather than a view of the world.
Cost without payoffAutomated filters catch most abuse, but terms touching tragedies, elections or named private people go to a reviewer before they show, with a target of under 5 minutes. Being a few minutes late on a sensitive trend is a far smaller problem than amplifying the wrong one.
07
Failure modes
Failure modes
Eight thousand accounts created over two weeks post the same tag within ten minutes. Raw counts would put it at number one in a country within two cycles.
Rank on distinct trusted authors, collapse near-identical text with SimHash, cap the share any account cluster (same creation week, same client, same IP range) can contribute, and alert reviewers when a term's author trust distribution looks unlike organic traffic.A goal in a World Cup final multiplies traffic on the global scope ten times for a few minutes. The worker that owns it lags, the ranker reads half-filled minutes, and the biggest moment of the year shows up late.
Split the global scope across several workers by term hash and merge their candidate tables at rank time. Use Kafka lag as a scaling signal, and have the ranker refuse to score a minute that is not closed, publishing the previous list with a stale flag instead.A morning story keeps its slot all evening because the score is stuck from a smoothed value that decays too slowly, while the actual minute counts have fallen back to normal.
Recompute the score from fresh counts every cycle and blend only the displayed value. Drop any trend whose acceleration has stayed below 1.0 for 30 minutes, regardless of its smoothed score.A term that violates policy, or a false claim about a tragedy, grows fast enough to trend before anyone has looked at it. Screenshots spread faster than the fix.
Block known terms at the extractor, send every new term touching sensitive entity classes to review before display, and give the suppression endpoint a path that purges Redis and the CDN in seconds.A popular account with millions of followers lists its location as one city but posts about another. Its followers' replies get counted in the wrong metro and a local trend appears somewhere nobody cares.
Prefer explicit place tags and recent location history over profile text, require a term to have authors from at least a few distinct sub-areas before it trends in a metro, and down-weight engagement replies compared with original posts.The nightly job silently writes nothing after a schema change. Baselines age, holidays are compared with ordinary days, and seasonal terms start trending every year.
Monitor the baseline table's freshness as a first-class metric, keep the last good baseline rather than overwriting it with empty output, and alarm when the share of candidates with no baseline climbs above its usual 15%.08
How to run the interview
How to run the interview
Define trending before counting anything. Say out loud that you rank on change against history, not on volume. Interviewers listen for this distinction in the first five minutes.
Show why exact counting does not fit. Multiply distinct terms by scopes by time buckets. The number you get is the argument for sketches and a candidate set.
Know how a Count-Min Sketch errs. It overestimates and never underestimates, with error tied to width and total volume. Be ready to size one on the whiteboard.
Count people, not posts. Distinct authors with a trust weight is the single best defence against manipulation and costs one HyperLogLog per candidate.
Handle cold start explicitly. A brand-new hashtag has no baseline. Explain the floor you use and why it stops tiny groups from producing huge scores.
Partition by scope, then deal with the hot one. Most scopes are small. The global scope during a final is not, so describe how you split it and merge candidates at rank time.
Bring moderation into the design. A suppression path that purges caches, and a review queue for sensitive categories, show that you know trends are a public surface with real consequences.
Keep the read path boring. Precompute per scope, cache for 30 seconds, return volume buckets. Spending interview time on serving a 10 MB dataset is a sign of poor prioritising.
09
Growing the design in stages
Growing the design in stages
Each step below is driven by a measured limit. Do not start with sketches if one database can still count everything.
An hourly query groups hashtags by country and compares them with the same hour last week. Crude, but good enough to learn what users click on.
A stream processor counts hashtags per country in one-minute windows. Baselines come from a daily batch job. Lists refresh every five minutes.
State size forces the move to Count-Min Sketches with a bounded candidate set. Ranking switches to distinct trusted authors after the first visible manipulation incident.
Lists exist for cities and languages. Co-occurrence clustering merges duplicate cards, and a review queue sits in front of sensitive categories.
The global scope is sharded across workers, "For you" blends lists per viewer, and baselines move to hour-of-week with holiday corrections. The single counting job from stage 2 is long gone.
Watch and read
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.