Worked design — newsfeed at scale

1# Illustrative interview math (heuristics — label as such)2dau = 100_000_0003posts_per_user_day = 0.24post_write_qps = dau * posts_per_user_day / 86_400   # ~230 QPS posts avg5peak_posts = post_write_qps * 10                     # event spikes67avg_followers = 2008# Pure push: each post fans out to ~200 timeline inserts9fanout_write_qps = post_write_qps * avg_followers   # ~46K QPS timeline writes1011# Home reads: 50% open feed × 10 reads/day12read_qps = dau * 0.5 * 10 / 86_400                  # ~6K avg → ~30K peak1314# Celebrity: 30M followers → one post = 30M writes — system death if naive push15# DECISIONS FORCED:16# 1) fan-out service decoupled via Kafka from post write ACK17# 2) celebrity detection on write → pull/merge path18# 3) timeline as IDs not full bodies; hydrate from post cache19# 4) inactive-user skip or lazy fan-out to cut write amp

1def on_post_created(post):2	followers = graph.followers(post.author_id)3	if is_celebrity(post.author_id):4		outbox.add(post.author_id, post.id)5		return6	# chunk fan-out to workers7	for chunk in chunks(followers, 1000):8		fanout_q.push(FanoutJob(post.id, chunk))910def home_timeline(user_id, cursor):11	ids = timeline_store.page(user_id, cursor, n=50)   # precomputed12	celeb_ids = merge_celeb_outboxes(user_id, cursor)  # pull path13	ids = merge_by_time(ids, celeb_ids)[:50]14	return hydrate(ids)  # post cache / multi-get1516FANOUT MODEL TRADEOFF17              Push (write)           Pull (read)            Hybrid18Read lat      very fast             heavy merge            mixed19Write cost    O(followers)          O(1)                   mixed20Storage       O(users×recent)       O(posts)               mixed21Freshness     bounded by fanout     live                   mixed22Pick          <~10k followers       celebs / sparse        production social

1POST /posts {text, media_refs[]} → {post_id}2POST /follow {target_user_id}3GET  /feed?cursor&limit=50 → {items[], next_cursor}4POST /like {post_id}   # eventual counter56# Cursor = (ts, post_id) for stable pagination under concurrent inserts7# Never use OFFSET for hot home feeds at scale

1FANOUT WORKER CHECKLIST2[ ] Idempotent writes to timeline3[ ] Chunked follower lists4[ ] Celebrity short-circuit before enqueue5[ ] Inactive-user skip policy6[ ] Lag metric + alert7[ ] DLQ + replay8[ ] Partial failure retry without duplicating entire fanout

1FEED DECISION LOG2Signal                         Forces3-----------------------------  ----------------------------------4Read:write >> 1                materialize or heavy cache on read5Power-law followers            hybrid / celebrity pull6Inactive majority              skip or lazy fanout7Viral post hydrate misses      single-flight post cache8Edits/deletes                  SoT filter on hydrate9Ranking uncertainty            chrono MVP + filters first1010× DAU                        storage medium + stricter skip1112Push cost example: 230 post QPS × 200 followers ≈ 46k timeline writes/s avg (heuristic).

1HEURISTIC WORKED EXAMPLE2DAU 1e8; posts/user/day 0.2 → post QPS ~230 avg3avg followers 200 → pure push ~46k timeline writes/s4home: 50% open * 10 reads → ~6k RPS avg → ~30k peak5celeb 3e7 followers → one post 3e7 writes → FORBIDDEN pure push6THEREFORE: hybrid; fanout service on Kafka; timeline IDs; hydrate cache78MODEL9posts(SoT) | follows | timeline(user,ts,post_id) | outbox(author,ts,post_id)10API: POST /posts, POST /follow, GET /feed?cursor1112FANOUT WORKER13chunk followers; idempotent (post_id, follower_id); lag SLO; DLQ14skip inactive; short-circuit celebrities before enqueue1516READ PATH17page timeline IDs → merge celeb outboxes → hydrate posts → filter deleted/mute18ranking: chrono MVP; ML later / companion track1920COST LEVERS21trim timeline length; skip inactive; CDN media; batch hydrate22page on fanout lag + home p99 + hydrate miss rate2324TRAPS25Kafka ≠ free fanout; full body denorm everywhere; sync delete N timelines