Design CDN

Two API surfaces: the data plane — HTTP(S) requests from end-users hitting edge PoPs — and the control plane — customer-facing API for configuring rules, purging cache, reading analytics.

Three tiers: edge PoPs (close to users, serve 90%+ of requests), regional tier (fewer, larger, act as a mid-cache + origin shield), and origin (customer's server). Requests cascade upward only on cache miss. A control plane in a few central regions handles config + purge + analytics rollup.

Anycast routing. All CDN PoPs announce the same IP address (typically a single /24 per service) via BGP. Internet routers naturally prefer the topologically closest announcement. A user in London gets routed to the London PoP; a user in Tokyo to the Tokyo PoP — without DNS games. If a PoP dies, its BGP announcement is withdrawn and traffic reroutes to the next-closest.

This is dramatically simpler than DNS-based geographic routing (which is the Akamai-era approach): no TTL games, no resolver location guesses, no DNS caching issues. Cloudflare popularized Anycast-for-CDN; now industry standard.

Cache hierarchy with tiered cache. Problem: 300 PoPs × 100 TB each is 30 PB of storage — but cumulative customer content is far more than that. Each PoP can only hold a subset. Result: cold content causes repeated fetches from origin, hammering customer servers.

Solution: regional tier between edge and origin. A PoP miss queries its regional tier. The regional tier has more storage, sees more traffic, and has better hit ratio. Only regional-tier misses hit origin. Net: origin load drops 10× compared to flat-edge.

sequenceDiagram
    participant U as User
    participant E as Edge PoP
    participant R as Regional
    participant O as Origin

    U->>E: GET /foo.jpg
    E->>E: cache lookup
    alt HIT (90% of the time)
        E-->>U: serve from NVMe (p50 ~5 ms)
    else MISS at edge
        E->>R: fetch /foo.jpg
        alt HIT at regional
            R-->>E: bytes + cache-control
            E-->>U: serve + populate edge cache
        else MISS at regional
            R->>O: fetch /foo.jpg
            O-->>R: bytes
            R-->>E: bytes
            E-->>U: serve + populate both caches
        end
    end
          

Cache key. Default is (host, URL-path, query-string). Customer can override — strip tracking params, normalize case, include Vary headers (Accept-Encoding, device type). Mis-configured cache keys are a top source of inexplicable "why isn't my site caching?" support tickets.

Purge / invalidation. Two flavors: URL-based (specific URLs) and tag-based (arbitrary label applied to responses at cache-time, then "purge all cached items with tag X"). Purge flow:

1. Customer API call hits control plane.
2. Control plane writes purge message to a global pub/sub bus (Kafka / internal multicast).
3. Every PoP subscribes; consumes message.
4. Each PoP invalidates matching entries in local cache (usually by writing a tombstone so serves return miss).

Total time: seconds. Not instant — but predictable. "Purge everything" zone-wide is much slower (hours) because it invalidates millions of keys.

Stampede prevention. If 1M users request a newly-popular image simultaneously and edge cache misses, all 1M requests would forward to the regional/origin. Protection: request coalescing — concurrent requests for the same key at a single PoP collapse into one upstream fetch. Only the first miss goes upstream; others wait for its response. One-request-in-flight invariant per key per PoP.

• Anycast vs DNS-based routing. Anycast is simpler, faster to fail over, no TTL dependencies. DNS-based allows policy ("send premium customers to better PoPs") but is brittle. Cloudflare-style = Anycast; Akamai = historically DNS. New CDNs default to Anycast.
• Flat-edge vs tiered cache. Flat (no regional) is simpler but hammers origin on misses. Tiered requires another hop but preserves origin. Modern default is tiered; "flat-edge" survives only in smallest deployments.
• Pull vs push cache. Pull = first request populates cache (lazy). Push = pre-populate before launch (eager, for known-popular events). Pull is default; push is opt-in for live-streaming Superbowl-style events.
• Shared vs per-customer caches. Shared cache (all customers share PoP capacity) is more efficient but creates noisy-neighbor risk. Per-customer quotas prevent one customer filling the cache. Some products isolate premium customers into dedicated capacity.
• TLS cert distribution. Every PoP needs the cert for every customer on it. Options: replicate all certs to all PoPs (wastes storage), fetch on-demand (slow first-request), or SNI-based lazy pull with LRU. Cloudflare uses combination of Keyless SSL + selective pre-stage.

1. Anycast first, not DNS. Modern CDN answer. Explain the mechanism in one sentence.
2. Tiered cache beats flat-edge. Without it, your origin burns. Mention by name; it's a load-reducing design choice that shows sophistication.
3. Purge is not magic. It's pub/sub with seconds of propagation. Don't say "instant cache invalidation."
4. Request coalescing for stampedes. One-request-in-flight per key. Classic pattern that many candidates forget.
5. Analytics + logs. CDN's real product is observability. Mention shipping edge logs → central analytics; it's half the value customers pay for.
6. Stale-while-revalidate. One line, massive impact. Shows you know the production-grade answer.