Communication
Design Reminder Alert
01
What reminders must do
What reminders must do
- Create a one-off reminder for a local date and time, such as "8 October, 09:00 in Asia/Kolkata".
- Create recurring reminders: every weekday at 07:30, the first Monday of each month, the last day of the month.
- Choose delivery channels per reminder: push, email or SMS, with a fallback order.
- Snooze, edit, pause or delete a reminder at any time, including seconds before it fires.
- Pick whether a reminder is pinned to a time zone or follows the device as the user travels.
- See history: when each occurrence fired, which channel delivered it and whether it was opened.
- Fire within 5 seconds of the intended instant for 99% of reminders, and within 60 seconds for all of them.
- Never skip an occurrence because of a deploy, a crash or a DST change.
- Users see each occurrence once; retries are invisible to them.
- Handle the top-of-the-hour spike, when a large share of all reminders fire in the same minute.
- Durable: an acknowledged reminder survives the loss of any single machine or zone.
Ask how precise "on time" has to be, and what should happen when the user changes time zone. A medication reminder at 08:00 probably means 08:00 wherever the user is; a meeting reminder means the meeting's zone. That single product decision changes how you store every reminder.
02
Sizing the system
Sizing the system
Assume 60 million monthly users, 450 million active reminders of which about 70% recur, and 140 million occurrences fired per day. Reminders cluster heavily on round times: about 2% of a day's fires land in the single busiest minute.
| Quantity | Working | Result |
|---|---|---|
| Fires per day | assumption | 140M |
| Average fire rate | 140M / 86,400 s | about 1,600 / s |
| Busiest minute | 2% of 140M | 2.8M fires, about 47,000 / s |
| Delivery attempts | 1.3 attempts per fire with retries and fallbacks | about 180M a day |
| Creates and edits | about 25M a day | about 290 / s, peak around 1,000 / s |
| Reminder definition | ids, title, rule, zone, channels, state | about 500 B |
| Definition storage | 450M x 500 B, 3 replicas | about 225 GB, 675 GB stored |
| Schedule index | one row per next occurrence, about 40 B | about 18 GB |
| Delivery history | 180M x 120 B a day, kept 90 days | about 2 TB |
The average rate is trivial. The busiest minute is about 30 times the average, and push providers rate-limit per app. The scheduler must read due work ahead of time and the delivery tier must be able to absorb a short burst without anything firing a minute late.
03
Endpoints
Endpoints
04
Architecture
Architecture
flowchart LR
App["Mobile and web apps"] --> API["Reminder API"]
API --> DB[("Reminder definitions")]
API --> SI[("Schedule index by minute bucket")]
SC["Scanners, one lease per shard"] --> SI
SC --> TW["In-memory timing wheel"]
TW --> Q[["Kafka: due fires"]]
Q --> DW["Delivery workers"]
DW --> DD[("Dedup keys in Redis")]
DW --> PN["APNs and FCM"]
DW --> EM["Email and SMS providers"]
DW -. "fired" .-> RM["Re-materialiser"]
RM --> SIKeep the user's intent and the machine's schedule apart. The definitions table stores what the user asked for: a wall-clock time, a zone and an optional recurrence rule. The schedule index stores only one thing per reminder, the next instant it should fire in UTC, filed under the minute it belongs to and a shard number. Everything that fires reads from the index; everything the user sees reads from definitions.
A pool of scanners splits the index into 256 shards and holds a lease per shard. Each scanner reads its shards a couple of minutes ahead, loads due entries into an in-memory timing wheel, and publishes each one to Kafka at its exact second. Delivery workers send through the right provider and record the result. After a fire, the re-materialiser computes the next occurrence from the rule and writes it back into the index.
05
Deep dive: buckets, time zones and delivering once
Deep dive: buckets, time zones and delivering once
Time buckets instead of a global scan
A query like WHERE next_fire <= now() over 450 million rows, run every few seconds, gets slower as the table grows and turns into a disaster at the top of the hour. Bucketing turns it into a lookup. Each schedule row is keyed by (minute_bucket, shard), where the shard is a hash of the reminder id modulo 256. To find everything due at 09:00, a scanner reads exactly the rows for that minute and its shards, and nothing else.
Scanners read two minutes ahead. For the 09:00 bucket, the read happens around 08:58, so the database load of the spike is spread over a quiet period. Entries go into a timing wheel with one slot per second, and the wheel emits them on time. Rows are marked "claimed by scanner N at lease epoch E" rather than deleted, which makes a crashed scanner's work recoverable.
flowchart TD
L["Hold lease on shard, epoch E"] --> R["Read bucket for now plus 2 minutes"]
R --> W["Load entries into timing wheel"]
W --> T{"Entry's second reached?"}
T -->|no| W
T -->|yes| V["Check version is still current"]
V --> P["Publish fire with key id plus instant"]
P --> M["Mark row emitted"]
M --> N["Advance to next bucket"]
N --> RLeases, not one leader
A single leader scanning everything is simple until it pauses for a long garbage collection at 08:59. Splitting work into shards with a lease each, kept in etcd or a similar store with a 10-second TTL, spreads load and limits blast radius. When a scanner dies, its leases expire, another scanner picks them up with a higher epoch and re-reads the current and previous bucket for those shards. Anything already marked emitted under the old epoch is skipped; anything only loaded is fired late by a few seconds rather than lost. The epoch also fences out the old owner if it comes back from a pause and tries to publish.
Time zones are the real bug factory
Store reminders as wall-clock time plus an IANA zone name, never as a UTC instant alone. "09:00 in New York" is 13:00 UTC in summer and 14:00 UTC in winter. If you only kept the UTC instant from the day it was created, every recurring reminder would drift by an hour twice a year.
The UTC instant in the schedule index is derived and disposable: it is computed from the rule and the zone each time the next occurrence is needed. That gives you clean answers to the awkward cases:
- A time that does not exist. When clocks spring forward from 02:00 to 03:00, a 02:30 reminder has no instant that day. Fire it at the first valid moment after, 03:00, and do not skip the day.
- A time that happens twice. When clocks fall back, 01:30 occurs twice. Fire on the first occurrence only, and make sure the recurrence engine does not schedule the second.
- The user travels. For reminders set to follow the device, a zone change on the phone triggers a recompute of every such reminder for that user. Pinned reminders are left alone.
- The rules change. Governments change DST rules with a few weeks' notice. When a new tz database release ships, recompute future instants for every reminder in an affected zone.
Expanding "every weekday at 07:30" into years of future rows multiplies storage and makes every edit a mass update. Keep the rule, store one upcoming instant, and compute the following one after each fire. An edit then touches exactly one index row.
At least once, shown once
Exactly-once delivery across a scanner, a queue, a worker and a push provider is not achievable, because a worker can always crash after the provider accepted a push and before it recorded success. The realistic contract is at-least-once sending with deduplication at every hop.
Each fire carries an idempotency key made from the reminder id and the occurrence instant. A delivery worker does SET key NX EX 86400 in Redis before calling the provider; if the key already exists, the fire is a duplicate and is dropped. Push payloads carry the same key as a collapse or thread id, so even if two pushes slip through, the phone shows one notification. Before sending, the worker also compares the fire's version with the definition's current version, which is how a delete or snooze made five seconds earlier wins over a fire already in the queue.
06
Common mistakes
Common mistakes
The offset between local time and UTC changes with daylight saving, so a "09:00 every day" reminder starts arriving at 08:00 or 10:00 for half the year.
The query cost grows with every reminder ever created and peaks at exactly the moments it must be fastest.
A deploy, a pause or a dead disk on that machine silently stops every reminder in the system.
A daily reminder becomes thousands of rows, a change to it rewrites all of them, and tz updates require rewriting the whole table.
No chain of networked components can guarantee it, so the promise ends up implemented as "maybe never" when something fails.
07
Decisions and their costs
Decisions and their costs
Database reads are predictable and proportional to what is due. Seconds-level accuracy comes from the in-memory timing wheel, not from the database.
Scales with due workSome brokers support delayed messages, which looks simpler. But reminders can be months away, edits and deletes mean finding and cancelling messages, and most brokers cap delay length. Fine for short delays such as snoozes.
For snoozes onlyEach scanner owns a slice and a failure affects only its shards for one lease period. Capacity grows by adding scanners.
No single point of failureEasy to reason about and fine for a few thousand fires a minute. At 47,000 per second, one process becomes both a throughput limit and a risk.
Good for early stagesNo occurrence is lost to a crash, and duplicates are removed by a cheap Redis check and the device's collapse id. The cost is a dedup store and some care with keys.
Honest guaranteeNo duplicates, ever, because nothing is retried. Also no reminder when a worker restarts at the wrong moment, which is the one failure users never forgive.
Loses remindersFor low-stakes reminders, add up to a few seconds of jitter so a minute's 2.8 million fires do not hit providers in the same second. Reminders flagged as time-critical skip the jitter.
Smooths provider loadA user may notice a reminder at 09:00:04. Making jitter opt-out per reminder, and keeping it within the 5-second target, keeps that acceptable.
Within the SLO08
Failure modes
Failure modes
It had loaded 60,000 entries into its wheel and emitted half. The other half exists only in its memory.
The lease expires within 10 seconds and a new owner re-reads the bucket. Rows marked emitted are skipped; the rest fire a few seconds late. Dedup keys catch any entry the old scanner published but did not mark.A government announces that clocks will not change this year. Every future instant computed with the old rules for that zone is now an hour wrong.
Ship tz database updates as a tracked release, then run a job that recomputes the next instant for every reminder in the affected zones. Because only one instant per reminder is stored, the job is small.The provider accepts the push, but the device has no signal for three hours and the reminder is pointless when it finally appears.
Set a push expiry that matches the reminder's usefulness, ask the client to acknowledge display, and fall back to email or SMS if no acknowledgement arrives within a few minutes for reminders marked important.A delivery worker sends a push, crashes before committing its Kafka offset, and a replacement processes the same message again.
The Redis dedup key set before the first send stops the second one. If Redis itself was lost, the push collapse id still merges the two on the device."Every month on the 31st" has no date in April. "Every year on 29 February" has none in most years. Naive date arithmetic skips or crashes.
Use a tested RRULE library, offer "last day of month" as its own rule, and define the behaviour for missing dates explicitly in the product, then test across the next twenty years of calendars.At 09:00 the delivery tier sends faster than the provider's per-app limit and starts getting rejections.
Keep long-lived HTTP/2 connections to providers, rate-limit per provider with a token bucket just under their limit, prioritise time-critical reminders, and retry rejected sends with short backoff.09
How to handle the interview
How to handle the interview
Lead with time zones. Say early that you store wall-clock time and a zone, and why storing UTC alone breaks recurring reminders. It is the single most common gap in answers to this question.
Turn scanning into lookups. Minute buckets with shards, read slightly ahead, show you understand why a range scan over everything collapses at the top of the hour.
Explain how a dead scanner is recovered. Expiring leases, epochs that fence out a paused owner, and re-reading the current bucket are what interviewers probe.
Be honest about delivery guarantees. State at-least-once plus dedup, and show where the dedup key comes from. Claiming exactly-once invites a hard follow-up.
Walk through the DST cases. The missing 02:30 and the repeated 01:30 each need a rule. Having one ready signals real-world experience.
Handle edits racing a fire. Versions on definitions, checked by the worker just before sending, settle the "I deleted it and it still buzzed" question.
10
How it grows
How it grows
A job runs every minute, selects due rows by an indexed timestamp and sends pushes. Zones and rules are stored properly from day one, because that part is painful to migrate later.
Firing and delivery split apart with a queue, delivery workers scale on their own, and dedup keys arrive with the first retry logic.
The schedule index is bucketed by minute and sharded, scanners take leases, and timing wheels give seconds-level precision.
Scanners and workers run in several regions with shard ownership per region, and delivery falls back from push to email or SMS for important reminders that are not acknowledged.
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.