Design Notification System

Build an internal service that other teams call to notify users. A caller says who should be told and what kind of event happened. The platform owns everything after that: looking up the user's devices and addresses, checking preferences, rendering the right template in the right language, choosing channels, sending through external providers, retrying failures and recording what happened.

The traffic is two very different things sharing one pipe. Transactional messages such as login codes and payment receipts are triggered by one user's action and must arrive within seconds. Bulk messages such as a marketing campaign or a breaking news alert target tens of millions of people at once and can tolerate minutes of spread. The design must make sure the second kind never delays the first.

Assume 120 million daily active users, an average of about 12 notifications each per day across channels, and a few large campaigns a week. Push dominates volume; SMS dominates cost.

The API does as little as possible on the request path: authenticate the caller, check the idempotency key, validate the payload against the event type's schema, write to an intake topic and return 202. Everything slow happens after that, so a provider outage never makes the API slow.

The router consumes intake events and turns each into zero or more channel deliveries. It loads the user's preferences and devices from a cache, applies opt-outs, quiet hours and frequency caps, renders the templates, and writes each delivery to the queue for that channel and priority. In-app items go straight to the inbox store, and an online user gets a live nudge over their WebSocket.

Each channel has its own worker pool that talks to external providers. Separate pools mean an email provider slowdown cannot back up push, and each pool can be rate limited to match its provider's contract. Workers record every outcome to a delivery log, which feeds the status API, analytics and the token cleanup job.

A campaign is not one big job; it is millions of small ones released at a controlled rate. The planner reads the segment in pages of, say, 10,000 users and emits intake events at a token-bucket rate chosen for the campaign. A 60 million push campaign at 50,000 a second takes 20 minutes, which is fine for a sale and avoids hitting provider limits.

Release it in steps. Send to 1 percent first and watch the error and unsubscribe rates for a few minutes. If something is wrong, such as a broken deep link or a template rendering "Hi {name}", the planner pauses automatically and the damage is limited to a small slice. Time zone aware campaigns are split into 24 or more buckets, each released when the local clock hits the target hour.

Because campaigns run in the low priority lane, the router and workers for critical traffic are unaffected. If the low lane backs up, messages wait; if they wait past their expiry, they are dropped as stale rather than arriving at midnight.

Every hop in this pipeline can fail after doing its work but before confirming it. The only safe policy is at-least-once: never acknowledge a message until its outcome is recorded, and accept that retries will happen. Duplicates are then removed with idempotency at three points.

At the API, the caller's key maps to the original notification ID for 24 hours, so a caller retrying a timeout creates nothing new. In the router, each delivery gets a deterministic ID such as a hash of notification ID, channel and device, so reprocessing an intake event yields the same deliveries. At the worker, a claim in a fast key-value store with a short lease marks the delivery as in progress before the provider call and as sent after it. A second worker that picks up the same delivery sees the claim and skips it.

There is still one gap: a worker can call the provider, the provider can send, and the worker can crash before writing "sent". When its lease expires, another worker retries. For email and some SMS gateways you can pass the delivery ID as the provider's own idempotency key and the duplicate is dropped there. For push, the platforms accept a collapse identifier, so a repeated push replaces the first on the device instead of showing twice. That brings the visible duplicate rate close to zero.

Classify provider responses before retrying. A throttling response or a 5xx means try again later, with exponential backoff and jitter. An invalid or unregistered token means the device is gone: disable the token and stop. A content rejection means the message will never succeed: fail it and alert the owning team. Retrying permanent errors wastes provider quota and can get your sender account flagged.

Each channel has a retry budget tied to the message's usefulness. A login code older than five minutes is worthless, so it expires; a receipt email can retry for a day.

The inbox is a per-user list stored in a wide-column table keyed by user ID and sorted by time, which makes "latest 20" a single partition read. Writes happen at send time, so reads never compute anything. The unread count lives as a small counter next to the list. When the user is online, the router also pushes a lightweight event over their existing app connection so the bell updates instantly; when offline, nothing extra is needed because the next open reads the list.