Designing a Scalable Notification Service

Sending one email is easy. Sending 10 million push notifications in 2 minutes for "Breaking News" is an architectural challenge.

1. Requirements

Functional

• Send: Support Email (SES), SMS (Twilio), Push (FCM/APNS).
• Bulk: Support "Broadcast" to all users.
• Preferences: Don't email me if I opted out.

Non-Functional

• Reliability: Never lose a notification.
• Rate Limiting: Don't get banned by Apple/Google/Twilio for spamming.
• Latency: Breaking news must be delivered fast.

2. High-Level Architecture

We need to decouple the Producer (The service wanting to send a message) from the Consumer (The worker calling the external API).

Components

1. Notification Service: The API gateway. Receives POST /send.
2. Message Queue (Kafka): Buffers requests. Prevents system crash if 1M requests hit at once.
3. Workers: Pull from Kafka and call external APIs (FCM/Twilio).
4. Rate Limiter: Controls the speed of workers.

3. The Message Queue (Buffer)

Why Kafka?

• Buffering: If Apple's configured limit is 10k/sec, but we receive 100k/sec requests, Kafka holds the backlog.
• Topics:
  • topic-high-priority (OTP codes, Login alerts) -> High number of consumers.
  • topic-low-priority (Marketing emails) -> Fewer consumers.

4. Reliability & Retry Mechanisms

External services (Twilio, FCM) fail all the time. What if workers call Twilio and get 500 Internal Server Error?

The Retry Queue

1. Worker fails to send email.
2. Push the message to a Retry Queue with a delay (Exponential Backoff).
3. Wait 1s, then 2s, then 4s, then 8s.
4. After 5 retries, move to Dead Letter Queue (DLQ) for human inspection.

5. Deduplication

• Problem: Retries might cause duplicate emails if the failure was a network timeout (the server actually sent it, but the ACK was lost).
• Fix: Check NotificationLog database.
  • INSERT INTO logs (id) VALUES (msg_id)
  • If insert fails (Duplicate Key), stop.

6. Rate Limiting (Token Bucket)

We must protect third-party quotas.

• Twilio Limit: 100 SMS/sec.
• Worker Logic:
  • Before sending, Worker asks Rate Limiter (Redis): "Can I take a token?"
  • If Redis says "Yes" (count < 100), proceed.
  • If Redis says "No", Worker sleeps or re-queues the message.

7. Preference Service

Before sending "Marketing Email" to User A:

1. Worker calls Preference Service.
2. Checks: "Did User A unsubscribe from Marketing?"
3. If yes, drop message silently.

8. Fan-Out: The Broadcast Problem

"Send to one user" and "send to all 50 million users" are different problems wearing the same API.

A naive broadcast — SELECT * FROM users followed by 50 million individual enqueues from one process — takes hours and melts the user database. Production systems split the job:

1. Chunking: the broadcast request is split into ranges (user IDs 0–99,999, 100,000–199,999, …) and each range becomes one Kafka message.
2. Parallel expansion: a fleet of fan-out workers consumes ranges, resolves each range to concrete device tokens/emails (with preference filtering applied here, in bulk), and emits individual send-jobs.
3. Progressive rollout: large broadcasts are often deliberately released in waves (1% → 10% → 100%) so that a bad notification — wrong link, broken deep-link, typo — can be halted before it reaches everyone. Treat notifications like deployments.

There is also a second-order effect to plan for: the self-inflicted thundering herd. Ten million push notifications saying "Your weekly report is ready" generate millions of app opens within minutes — hitting your own API servers. Coordinate broadcast schedules with capacity planning, or jitter delivery across an hour.

9. Delivery Tracking and Observability

"Never lose a notification" is only provable if you track state transitions. A common lifecycle model:

CREATED → ENQUEUED → SENT → DELIVERED → OPENED
                      ↓
                   FAILED → RETRYING → DEAD-LETTERED

• SENT means the third-party provider accepted the request (Twilio returned 200).
• DELIVERED requires provider callbacks/webhooks (Twilio status callbacks, APNS feedback, SES bounce notifications) — wire these into the same log.
• Bounce and complaint handling is not optional for email: providers track your bounce rate, and exceeding a few percent gets your sending domain throttled or blacklisted. Hard bounces must automatically suppress the address from future sends.

Key metrics to alert on: end-to-end delivery latency per channel (p50/p99), DLQ arrival rate (a spike means a provider or template is broken), provider error rates per region, and opt-out rate per campaign (an early-warning signal that content is being flagged as spam).

10. Idempotency Keys Done Right

The deduplication insert in section 5 works, but the key deserves thought. Deduplicating on a random message UUID only protects against queue redelivery. It does not protect against the upstream service retrying its own call and generating a new UUID for the same logical event.

The robust pattern: idempotency keys derived from business semantics — user_42:order_981:order_shipped. Now any path that tries to tell user 42 about order 981 shipping collides with the first attempt, regardless of which service retried or when. Give the log table a TTL (say, 7 days) so it doesn't grow forever; the deduplication window only needs to exceed your maximum retry horizon.

Interview Checklist

When this design comes up in a system design interview, the strongest signals you can send:

• Draw the queue before being asked about spikes — decoupling is the core of the design.
• Mention priority isolation: OTP codes must never sit behind 10M marketing messages. Separate topics and separate worker pools (a shared pool with priorities still suffers head-of-line blocking during a marketing blast).
• Distinguish at-least-once delivery + idempotent consumers from "exactly-once," and explain why the former is what everyone actually builds.
• Bring up provider abstraction: wrap Twilio/FCM/SES behind an internal interface so a provider outage becomes a config change (failover to a second SMS provider), not an incident.

Summary

1. Decouple: Use Queues (Kafka/RabbitMQ) to absorb spikes.
2. Sort: Prioritize OTPs over Marketing.
3. Protect: Rate limit your workers to respect external API limits.
4. Retry: Use exponential backoff for resilience — and dead-letter what keeps failing.
5. Track: Model the full delivery lifecycle so "sent" and "delivered" are never confused.

Related Concepts

• Message Queues
• Kafka Architecture
• Rate Limiting
• Token Bucket Algorithm
• Backpressure