Want to try it yourself first?
Answer each stage in the practice workspace, sketch the architecture, or run a timed mock, then come back here to compare.
The problem
A notification system takes a request like "tell these users their order shipped" or "tell all 10 million users about the sale" and turns it into push notifications, text messages and emails, through third-party providers that are slow, rate-limited and occasionally down. Sending one message is trivial. The hard parts are respecting each user's preferences, fanning a single broadcast out to millions of devices in minutes, retrying failures without hammering providers, and making sure a retry never sends the same notification to the same person twice.
Watch: Design a Notification System
The API stores the request with a caller-chosen notification_id and returns 202. Fan-out workers read the audience in batches of 1,000, skip users who opted out, are in quiet hours or hit their daily limit, and put one message per channel on its own queue, with transactional traffic on separate queues. Workers ack only after sending, retry timeouts, 5xx and 429 with exponential backoff and jitter, and dead-letter after 5 attempts. A Redis SET NX on notification_id:user_id:channel stops redelivered duplicates, provider collapse keys cover a crash right after sending, dead tokens (APNs 410, FCM UNREGISTERED) are deleted, and a token bucket keeps SMS under the gateway's limit.
Functional Requirements
The core features we need to support:
- Send to one user (transactional): "your order shipped", "your password was changed".
- Send to many users (broadcast): a sale announcement to 10 million users, or to a segment like "users in Germany who haven't opened the app in 30 days".
- Three channels: mobile push (APNs for iOS, FCM for Android), SMS, and email.
- Respect user preferences: per-channel and per-category opt-outs ("no marketing email"), quiet hours in the user's time zone, and a per-user rate limit (e.g. at most 3 marketing pushes a day).
- Templates: callers send a template id and variables, not raw text, so copy and translations live in one place.
- No duplicates: a user must never get the same notification twice on the same channel, even when the system retries.
- Delivery tracking: sent, delivered, opened, failed, per notification.
Example. The orders service calls the API with template = order_shipped, user_id = 42, notification_id = n-981. User 42 has push and email on, SMS off. The system enqueues one push and one email; each is sent once, and the push worker records "sent" against n-981:42:push.
Out of scope for a first pass: in-app notification inbox, A/B testing of copy, and rich media. Mention them so the interviewer knows you're scoping on purpose.
Scale Requirements
Rough numbers to anchor the design:
- 10 million push notifications for a broadcast, delivered within 10 minutes.
- 10 minutes = 600 seconds → 10,000,000 / 600 ≈ 17,000 pushes/s at peak (assume one device per user for the estimate; real fan-out to multiple devices multiplies this).
- Background traffic: ~100 million transactional notifications a day → 100M / 86,400 ≈ ~1,200/s on average.
- Fan-out: reading the audience in batches of 1,000 users → 10,000 batch jobs per broadcast.
- Workers: if one push worker sustains ~500 sends/s over HTTP/2 connections to the provider, 17,000 / 500 ≈ 34 workers; run ~40 for headroom.
- Dedup store: one key per (notification, user, channel). A key plus Redis overhead is ~100 bytes, so 10M keys ≈ 1 GB of memory, expiring after 24 hours.
The takeaway: 17,000 messages a second is a normal queue workload. The design is dominated by provider limits (APNs, FCM and especially SMS gateways throttle you), retries, and not sending duplicates while doing all of it at least once.
Non-Functional Requirements
- No duplicates. Getting the same push twice is the most visible bug a notification system can have. Treat it as a hard requirement.
- No silent losses. Every accepted notification is either sent, deliberately skipped (opted out, quiet hours), or ends in the dead-letter queue where someone can see it.
- Priorities. A password-reset SMS must not wait behind a 10-million-user marketing blast. Transactional and bulk traffic go through separate queues.
- Latency. Transactional: seconds. Broadcast: minutes is fine.
- Isolation. If the SMS provider is down, push and email keep flowing.
- Exactly-once is impossible end to end. Once a request reaches APNs, we can't take it back. The goal is at-least-once processing + idempotent sending, which is effectively once in practice.
API
- The API returns 202 Accepted right away; sending happens asynchronously.
notification_idmakes the API itself idempotent: if the caller retries the POST after a timeout, the second request is recognized and not enqueued again.- Device registration is a separate call from the apps:
PUT /v1/devices { user_id, platform, token }.
Data Model
| Store | Shape | Notes |
|---|---|---|
| Notifications | notification_id → template, variables, audience, category, status | one row per API request |
| Preferences | user_id → per-channel / per-category opt-ins, time zone, quiet hours | read on every send; cached |
| Device tokens | user_id → list of (token, platform, last_seen) | a user can have several devices |
| Templates | template_id + locale → title/body per channel | versioned |
| Rate-limit counters | user_id:category:day → count | Redis INCR with a TTL |
| Dedup store | notification_id:user_id:channel → sent | Redis SET NX with a 24 h TTL |
| Queues | push / SMS / email, each split into transactional and bulk | Kafka or SQS |
| Dead-letter queue | messages that failed all retries | inspected and replayed by operators |
| Delivery events | (notification_id, user_id, channel, event, time) | append-only, feeds analytics |
High-Level Design
The pipeline:
- A service calls the notification API. It validates the request, stores it, and returns 202.
- For a single user, the API enqueues one job. For a segment, it enqueues a fan-out job.
- Fan-out workers read the audience in batches of 1,000 user ids and, for each user, check preferences, quiet hours and the rate limit, then render the template and enqueue one message per allowed channel onto the per-channel queue.
- Channel workers take a message, check the dedup store, look up device tokens (for push), and call the provider.
- Success → mark sent in the dedup store and emit a delivery event. Retryable failure → retry queue with backoff. Permanent failure → dead-letter queue (or, for a dead token, delete the token).
- Provider callbacks and app open events flow into delivery tracking.
Each stage only talks to the next through a queue, so a slow SMS gateway backs up the SMS queue and nothing else.
Deep Dive: Fanning Out to 10 Million Users
Don't expand the audience in the API request. The API stores the broadcast and enqueues one fan-out job.
Split into batches. A coordinator pages through the segment (a precomputed list of user ids, or a query over a user-attributes table) and enqueues 10,000 batch jobs of 1,000 user ids each. Batch jobs are independent, so dozens of fan-out workers process them in parallel, and a crashed worker only redoes its own batch.
Per batch, per user:
- Load preferences for all 1,000 users in one bulk read (and from cache), not 1,000 lookups.
- Drop users who opted out of this category or channel. Users in quiet hours get a delayed message scheduled for their morning instead of a send now.
- Increment the per-user rate-limit counter; skip if over the cap.
- Render the template and enqueue one message per allowed channel.
Throughput math. 10,000 batches in 10 minutes is ~17 batches/s; each batch produces ~1,000 messages, matching the ~17,000 sends/s target. Fan-out runs on the bulk queues, so transactional messages are never stuck behind it.
Pacing. Pushing 10M messages onto the queue in 30 seconds is fine; the channel workers drain at the providers' allowed rate. The queue is the buffer.
Deep Dive: At-Least-Once Delivery, Retries and the Dead-Letter Queue
At-least-once. A worker acknowledges a queue message only after it has finished with it. If the worker crashes first, the queue redelivers the message to another worker. Nothing is lost, but the same message can be processed twice, which is why the next section exists.
Classify the failure:
- Retryable: timeouts, 5xx,
429 Too Many Requests, connection resets. - Permanent: invalid token, malformed request, phone number that can't receive SMS. Retrying won't help.
Exponential backoff with jitter. Retry after 1 s, 2 s, 4 s, 8 s, 16 s (doubling each time), and add random jitter to each delay, e.g. a random value between 0 and the computed delay. Without jitter, the 50,000 messages that failed together at 12:00:00 all retry together at 12:00:01 and knock the provider over again. If the provider sends Retry-After, honor it instead.
Dead-letter queue. After about 5 attempts, move the message to a DLQ with its last error. Operators get an alert when the DLQ grows, can inspect the messages, and can replay them once the cause is fixed. Replayed messages still go through the dedup check, so a replay can't create duplicates.
Circuit breaker. If a provider's error rate spikes (say 50% over a minute), stop calling it for a short time and let messages wait in the queue, rather than burning all their retries in a few seconds.
Deep Dive: Never Sending the Same Notification Twice
The idempotency key is notification_id + user_id + channel, e.g. n-981:42:push. It is the same no matter how many times the message is delivered or retried.
Check before sending with Redis SET NX:
NXmeans "only set if the key doesn't exist", so it's an atomic claim: two workers racing on the same message can't both get OK.- The short 60 s TTL on
"sending"covers a worker that crashes before calling the provider: the claim expires and a retry can try again. - After success, the key is upgraded to
"sent"with a 24 h TTL, long enough to cover every retry and DLQ replay window. At ~100 bytes per key, a 10M broadcast costs about 1 GB.
The remaining gap. If a worker crashes after the provider accepted the push but before writing "sent", the 60 s claim expires and the message can be sent again. To shrink that window, set the provider's own collapse key (apns-collapse-id on APNs, collapse_key on FCM) to the notification id, so a duplicate replaces the first one on the device instead of showing twice. For email and SMS, many providers also accept an idempotency key on their API.
The dedup store is also why the API's notification_id comes from the caller: a caller retrying its POST reuses the same id, and every downstream key is the same.
Deep Dive: Device Tokens and Provider Limits
Device tokens. The app registers its push token on every launch (PUT /v1/devices), updating last_seen. A user with a phone and a tablet has two tokens, so one push notification becomes two sends.
Invalid token cleanup. When a user uninstalls, the token goes dead. The provider tells you on the next send: APNs returns 410 Unregistered, FCM returns UNREGISTERED. Delete the token immediately and don't retry. Also expire tokens not seen in a few months. Dead tokens waste capacity and providers may throttle senders with high invalid-token rates.
Provider rate limits.
- Push providers accept high volume, but over a limited number of connections; workers keep long-lived HTTP/2 connections and multiplex many requests on each.
- SMS is the tightest: gateways cap messages per second per sender number, and each SMS costs money. A token-bucket limiter per provider account sits in front of the SMS workers, and the SMS queue simply grows during bursts.
- Email providers limit sends per second and watch bounce and complaint rates; hard bounces are removed from the list like dead tokens.
Fallback. For critical transactional messages (2FA codes), a second SMS provider can take over when the first is failing.
Deep Dive: Delivery Tracking and Analytics
- Every stage emits an event to an append-only stream: accepted, skipped (with reason: opted out, quiet hours, rate limited), sent, failed, dead-lettered.
- Providers report later states asynchronously: email delivered/bounced/opened via webhooks, SMS delivery receipts. Push delivery to the device is mostly invisible; opened comes from the app itself when the user taps the notification.
- A stream processor aggregates events into counters per
notification_idand channel, so the status endpoint reads a handful of numbers instead of scanning 10 million rows. - Raw events land in a warehouse for funnels ("sent → opened → purchased") and to tune rate limits and send times.
Wrap-Up & Trade-offs
A solid answer hits these points:
- Pipeline: API (202, caller-provided
notification_id) → fan-out → per-channel queues → channel workers → providers. - Preferences first: opt-outs, quiet hours in the user's time zone, per-user rate limits, and templates.
- Fan-out: a 10M broadcast becomes 10,000 batches of 1,000 users; 10M in 10 minutes ≈ 17,000 sends/s, on bulk queues separate from transactional traffic.
- At-least-once queues + retries at 1 s, 2 s, 4 s… with jitter, then a dead-letter queue after ~5 attempts.
- Idempotency:
notification_id + user_id + channelclaimed with Redis SET NX and a 24 h TTL, plus provider collapse keys for the crash-after-send gap. - Device tokens: delete on 410 / UNREGISTERED; provider rate limits enforced with token buckets.
- Tracking: an event stream aggregated per notification.
Trade-offs. Even an exactly-once queue can't stop a duplicate caused by a crash between the provider call and the acknowledgement, because the provider is outside the transaction; the dedup check is what actually protects users. Longer dedup TTLs are safer but cost memory. Strict per-user rate limits protect users but can drop a message they actually wanted, which is why transactional messages bypass marketing caps.
Common follow-ups: an in-app inbox (store notifications per user, read on app open), per-user send-time optimization, and multi-region deployment (dedup keys must live in one region per user, or be replicated).
Common follow-up
Halfway through a 10-million-user broadcast, FCM starts returning errors for 20% of requests and one of your push workers crashes after sending but before acknowledging its batch. Walk through what happens to the affected users, and how you guarantee none of them gets the notification twice.