How would you design 'Hot Topics' on Instagram?

To design Instagram's 'Hot Topics,' I'd focus on a scalable, real-time system.

1. Data Ingestion: Utilize Kafka to ingest posts tagged with hashtags. Each message would include the hashtag, timestamp, user ID, and post ID. Kafka's partitioning would handle high throughput and provide fault tolerance.

2. Real-time Processing: Employ a stream processing engine like Apache Flink. This would process incoming posts in real-time, applying sliding window aggregations (e.g., 1-hour, 24-hour, 7-day windows).

3. Topic Scoring: Calculate topic popularity using a weighted scoring model. This model would consider:

   • The sheer volume of posts for a hashtag.
   • User engagement metrics (likes, comments, shares).
   • The diversity of users engaging with the topic (unique user count).

4. Ranking and Serving: Store the calculated scores and rankings in a low-latency database (like Redis or Cassandra) for quick retrieval. A separate service would then serve these trending topics to users.