Design Amazon S3

Two distinct planes. The index plane (metadata: (bucket, key) → physical locations) is a sharded, highly-available KV service. The data plane (actual bytes, erasure-coded across disks and AZs) is the storage fleet. A request front-end authenticates + routes; a background garbage collector reclaims deleted bytes after safe intervals.

S3 promises "eleven nines" of durability. That's not a marketing line — it's a budget. 99.999999999% means if you store 10B objects you expect to lose ~1 per year. Achieving this cheaply is the central engineering challenge.

Triple replication is too expensive. Storing 3 copies gives ~6 nines and costs 3× storage. For exabyte scale, that's billions of dollars of overhead.

Reed-Solomon erasure coding. Split each object into 10 data shards; compute 4 parity shards. Any 10 of 14 can reconstruct the whole object. Losing 4 independent disks simultaneously (astronomically unlikely if spread across AZs) is required to lose data. Storage overhead: 1.4× instead of 3×. Durability: ~11 nines when placed across multiple AZs.

sequenceDiagram
    participant C as Client
    participant FE as Front-end
    participant IX as Index svc
    participant ENC as RS Encoder
    participant S as Storage fleet

    C->>FE: PUT /bucket/key (bytes)
    FE->>IX: allocate object_id
    IX-->>FE: object_id + partition hint
    FE->>ENC: split + encode bytes
    ENC->>ENC: produce 10 data + 4 parity shards
    par write across AZs
        ENC->>S: shard 1→10 (AZ a/b/c/d)
        ENC->>S: parity 1→4
    end
    S-->>FE: all 14 acked
    FE->>IX: commit {object_id, shard_locations, etag}
    IX-->>FE: committed
    FE-->>C: 200 OK + ETag
          

Hot partition problem. The metadata index is sharded. Historically S3 sharded by a hash of the full key, but ALSO used the key prefix for list performance. A naming pattern like 2026/04/12/log-XXX.json meant all today's writes hit the same few metadata partitions. Throughput caps at ~5500 PUT/sec per prefix; worst-case customers saw throttling.

S3 solved this in two steps:

1. Hot partition auto-split. Metadata service detects rising QPS on a partition and automatically splits it into two, rebalancing keys. Background process; live traffic largely unaffected.
2. Auto-partitioning (2018 launch). The service learned to auto-shard by key prefix entropy. Today's customers don't need to add random hash prefixes to their key names — S3 handles it. But for extreme rates, the old advice still helps: ${random_hex}/2026/04/12/log.json spreads load evenly.

Rebuild storm. When a disk fails, the system re-reads 10 of the 13 surviving shards per affected object to rebuild the lost shard onto a new disk. That's 10× traffic spike on the surviving disks during rebuild. Mitigations: prioritize low-redundancy objects, throttle rebuild bandwidth to prevent starving live traffic, and keep spare capacity so even a major failure doesn't push the system into the red zone.

Multipart upload for big objects. A 5 TB object can't reasonably be one PUT. Multipart lets you:

• Upload parts in parallel (higher throughput).
• Resume on failure (only re-upload the failed part).
• Abort to reclaim space (lifecycle policies clean stale uploads).

Each part is stored as its own erasure-coded blob; the final CompleteMultipartUpload just writes a manifest. No byte-level stitching required.

• Erasure coding vs replication. EC saves storage but costs more CPU (encode/decode on every read/write) and higher recovery bandwidth. Worth it only at large scale. Small-scale systems stick with replication.
• Strong vs eventual consistency. S3 became strong read-after-write in 2020. Before: writes could take minutes to propagate to all index replicas, leading to "missing object" errors right after upload. Now: linearizable on new objects, eventual only on list operations. Users pay nothing extra — AWS absorbed the engineering cost.
• Storage tiering. Hot (SSD), warm (HDD), cold (Glacier = tape / slow disk). 10× cost difference between tiers. Lifecycle policies auto-migrate; reads from cold tiers take minutes-to-hours. Important to expose as an explicit product rather than hiding.
• Per-object metadata overhead. Every object has a few KB of metadata (ACL, storage class, lifecycle markers). At 280T objects, that's ~1+ PB of metadata alone — not ignorable.
• Eventual consistency on list. List operations scan metadata partitions and may miss just-written objects for seconds. Users who need "list includes everything I wrote" either wait a few seconds or call GetObject directly.

1. Durability math is the core. 11 nines is a number. Work backward from it: what does that imply about replication factor, geographic spread, verification?
2. Name erasure coding by type. "Reed-Solomon (10, 4)" or "(12, 4)" is much more credible than "we use erasure coding." Know the storage-vs-durability tradeoff by heart.
3. Split the two planes. Index service (metadata) vs data fleet (bytes). Don't muddle them. The index is transactional; bytes are immutable blobs.
4. Hot partitions are the classic follow-up. Interviewer will ask "what if everyone's key starts with today's date?" Have the auto-split + hash-prefix answer ready.
5. Multipart upload is more than chunking. It's resumable, parallelizable, and lets you abort partial work. Describe each of those benefits distinctly.