Storage & Data
Design Amazon S3
01
What the store promises
What the store promises
- Create buckets; put, get and delete objects by bucket and key, from a few bytes up to 5 TB.
- Ranged reads, so a video player or a query engine can fetch just the bytes it needs.
- Multipart upload for large objects, with parts sent in parallel and retried individually.
- List keys in order under a prefix, with a delimiter to imitate folders.
- Optional versioning, lifecycle rules (expire, move to colder tiers) and write-once retention locks.
- Access policies per bucket and per key, plus time-limited signed URLs.
- Durability target of eleven nines per object per year: lose at most one object in 100 billion.
- Survive the loss of an entire availability zone without losing data or the ability to read.
- Strong read-after-write consistency: a successful PUT is visible to every later GET and LIST.
- Availability of 99.99% for reads and writes.
- First-byte latency in the tens of milliseconds; throughput limited by the client, not the store.
- Cost per stored gigabyte as low as possible, since storage, not compute, is the bill.
Ask whether the interviewer wants the data plane (bytes and durability), the metadata plane (keys, listing and consistency), or both. Also ask about the object size mix. Billions of 4 KB thumbnails and a few thousand 2 TB backups stress completely different parts of the design.
02
Back-of-envelope numbers
Back-of-envelope numbers
Design one large region. Assume 40 trillion objects totalling 600 PB of user data, a median object of 30 KB but a mean of about 15 MB because of large files, and 1.5 million requests a second on average, 80% of them reads.
| Quantity | Working | Result |
|---|---|---|
| Logical data | assumption | 600 PB |
| Raw capacity with 9+6 erasure coding | 600 PB x 15 / 9 | 1 EB |
| Raw capacity with 3 replicas instead | 600 PB x 3 | 1.8 EB |
| Drives needed | 1 EB / 24 TB drives, at 80% fill | about 52,000 drives |
| Drive failures | 52,000 x 1.2% annual failure rate | about 620 a year, nearly 2 a day |
| Requests | 1.5M / s average, 2.5x peak | about 3.75M / s at peak |
| Metadata records | 40T objects x about 250 B | about 10 PB, replicated 3 ways |
| Ingest | 300K PUT / s x mean written size 2 MB | about 600 GB / s |
| Rebuild per failed drive | 19 TB used, read from many peers in parallel | under an hour when spread wide |
Erasure coding saves 800 PB of raw disk against triple replication at this size, which is tens of thousands of drives. And with almost two drives dying every day, failure is not an incident, it is a background process that must run continuously and automatically.
03
The interface
The interface
04
Architecture
Architecture
flowchart LR
C["Client SDK"] --> LB["Load balancers"]
LB --> FE["Front-end API fleet"]
FE --> AU["Auth and policy"]
FE --> IX["Key index: range partitions, Raft groups"]
FE --> PL["Placement service"]
FE --> SN["Storage nodes"]
PL --> SN
SN --> AZ1[("Zone A disks")]
SN --> AZ2[("Zone B disks")]
SN --> AZ3[("Zone C disks")]
SC["Scrubbers"] --> SN
RB["Repair workers"] --> SN
GC["Garbage collector"] --> IXSplit the system into a metadata plane that knows where things are and a data plane that holds bytes. Front ends are stateless: they authenticate the request, check policy, talk to the key index and stream bytes to or from storage nodes. They never buffer an entire large object.
The key index maps (bucket, key, version) to a list of locations and checksums. It is partitioned by key range, each partition is a small replicated state machine using Raft across three zones, and partitions split and move as they grow or get hot. The placement service picks which storage nodes receive new data, balancing fill level, zone spread and drive health. The data plane spans three zones, and background fleets of scrubbers, repair workers and garbage collectors keep it healthy without any request ever waiting on them.
Storage nodes do not keep one file per object. They append objects into large extents of about 1 GB, so tiny objects do not waste blocks and erasure coding works on big, uniform units.
The client sends a checksum, the front end checks it, each fragment stores its own, and every read verifies it. Corruption is caught at the first byte that is wrong, not by a customer.
Scrubbers read every fragment on a cycle of a few weeks, repair workers rebuild what is missing, and garbage collection reclaims space from deleted and overwritten objects.
05
Deep dive: durability, consistency and hot prefixes
Deep dive: durability, consistency and hot prefixes
How a PUT becomes durable
flowchart TD P["PUT arrives at front end"] --> CK["Verify checksum while streaming"] CK --> OP["Append to an open extent, 3 replicas in 3 zones"] OP --> AK["All replicas fsync and acknowledge"] AK --> IX["Commit location to key index via Raft"] IX --> OK["Return 200 to client"] OP --> SE["Extent fills and is sealed"] SE --> EC["Background: encode into 9 data plus 6 parity fragments"] EC --> SP["Place 5 fragments in each zone"] SP --> FR["Free the 3 full replicas"]
New data first lands in an open extent replicated three ways, one copy per zone. Replication is the right choice while an extent is being appended to, because it acknowledges quickly and needs no computation. Only once all three replicas have written to stable storage does the front end commit the object's location to the key index and answer the client.
When an extent fills, it is sealed and becomes immutable. A background job then erasure-codes it into 9 data fragments and 6 parity fragments using Reed-Solomon, writes five fragments to each zone, and deletes the full replicas. Any 9 of the 15 fragments are enough to reconstruct the extent. Losing an entire zone takes out 5, leaving 10, so reads keep working; on top of that, any one further drive can fail before the data is at risk.
The durability arithmetic
Data is lost only if seven fragments of one extent fail before repair finishes. With fragments of each extent spread across different drives, racks and zones, those failures have to be independent and close together in time. If a drive fails at 1.2% a year and repair of its fragments completes within an hour because thousands of peers each rebuild a small piece, the window in which more failures can stack up is tiny. The probability of seven overlapping failures in the same group of 15 drives within that window works out to far beyond eleven nines.
In practice the arithmetic is not what loses data. Correlated failures are: a bad firmware release, a batch of drives from one factory lot, a software bug in repair, or an operator command. That is why placement spreads each extent across hardware generations and suppliers, why changes roll out one zone at a time, and why repair and deletion code get the most careful review in the whole system.
If a failed drive's data were rebuilt onto a single replacement drive, it would take more than a day at sequential speeds, and a day is a long window for a second and third failure. Spreading each drive's fragments across thousands of extents lets thousands of drives each rebuild a small slice in parallel, which is how the window shrinks to minutes.
Strong consistency in the index
Read-after-write consistency is a property of the metadata plane, not the bytes. Bytes are immutable: an overwrite writes new bytes and then swaps the pointer. Each index partition is a Raft group, so a committed write is on a majority of replicas in different zones before it is acknowledged, and reads are served by the leader or by followers that have confirmed they are caught up.
The hard part is caching. Front ends cache index entries heavily to keep latency low, and a cached entry can be stale. Each partition therefore keeps a monotonic version counter. A front end that serves from cache first checks, cheaply and in batches, that the partition's version has not moved past the one its cache entry was read at. That keeps strong consistency without sending every read to the Raft leader.
Hot prefixes
Because the index is partitioned by key range, keys that share a prefix live together. That is what makes LIST cheap, and it is also a trap. A logging pipeline that writes 2026-10-01/host-0001.log, 2026-10-01/host-0002.log and so on sends all of today's writes to the one partition holding today's date.
The index handles this by splitting partitions on load as well as size. A partition that sustains more than its share of requests splits at the median key of recent traffic, and the halves move to different hosts. Splitting takes seconds to minutes, so a sudden ramp can see throttling first, which the API reports as a slow-down error that SDKs retry with backoff. For workloads that write strictly increasing keys, the advice is to spread them, for example by putting a short hash or a shard number at the front of the key, at the cost of making a single prefix LIST less convenient.
06
Decisions and their costs
Decisions and their costs
Writes get the low latency of replication; data at rest gets the 1.67x overhead of 9+6 coding instead of 3x. The cost is a background encoding pipeline and more complicated reads during repair.
Best cost per durable byteSimple and fast to rebuild, but 800 PB more raw disk in this region alone. Still sensible for small hot datasets and for the metadata plane.
Too expensive at restOrdered keys make LIST and prefix queries natural, and partitions can split wherever traffic concentrates.
Supports ordered listingSpreads sequential keys evenly with no hot spots. But every LIST becomes a scatter-gather across all partitions, which is unacceptable at trillions of keys.
Breaks listingApplications can write a file and immediately read or list it, which removes a whole category of bugs from data pipelines. It costs consensus on every metadata write and careful cache validation.
What customers expect nowCheaper and easier to cache, and early object stores shipped it. Customers then built extra metadata services just to know whether their own writes had landed.
Legacy trade-offSmall objects pack together, coding works on uniform 1 GB units and drives see large sequential writes. Deleting an object only marks space free until compaction rewrites the extent.
Efficient for mixed sizesEasy to build first. Trillions of tiny files exhaust inodes, fragment disks and make erasure coding of small objects absurdly wasteful.
Fails on small objects07
Failure modes
Failure modes
A rack loses power and comes back with several drives dead, all holding fragments of overlapping extents. Some extents are now down to 10 or 11 of 15 fragments.
Prioritise repair by risk: extents with the fewest surviving fragments rebuild first, ahead of everything else. Placement already ensures no rack holds more than one fragment of an extent, so a rack failure costs at most one fragment per extent.An analytics job starts writing 40,000 objects a second under a single date prefix. One index partition saturates and requests start failing with slow-down errors.
Split partitions on sustained load, return explicit throttling errors that SDKs retry with jittered backoff, and publish guidance on key design so heavy writers spread their prefixes.A drive returns wrong bytes without reporting an error, or a memory fault flips a bit during transfer. Nothing notices until someone reads the object months later.
Store a checksum with every fragment and verify it on every read and during scrubbing. A mismatch triggers reconstruction from the other fragments, and the drive gets flagged for retirement.A policy edit grants read access to everyone, exposing customer records. The storage is working exactly as configured.
Block public access by default at the account level, require an explicit extra setting to override it, and alert owners when a policy change makes data world-readable.Clients start large uploads and crash before completing them. The parts are stored and billed but never appear as objects, and nobody knows they exist.
Expose incomplete uploads in listings and bills, and support lifecycle rules that abort uploads older than a few days automatically.A control plane bug or a large network event makes a region unreachable. Data is intact, but customers cannot read it.
Offer asynchronous cross-region replication for buckets that need it, keep regions fully independent so failures do not spread, and make the client SDK able to fail over to a replica bucket.08
How to handle the interview
How to handle the interview
Separate metadata from data in the first diagram. Saying that the key index and the byte store scale differently, and fail differently, frames every later decision.
Do the replication versus erasure coding sum. Show the raw capacity for both at your scale. Then explain why new writes are replicated first and encoded later.
Explain durability as repair speed versus failure rate. Wide placement and parallel rebuild shrink the window; correlated failures are the real threat.
Place consistency in the index. Immutable bytes plus a consensus-backed pointer give read-after-write. Mention how caches stay correct.
Bring up hot prefixes before you are asked. Date-prefixed keys are the textbook case, and splitting on load is the answer interviewers want to hear.
Remember the operational fleets. Scrubbing, repair and garbage collection are where much of the engineering lives, and naming them signals real experience.
09
How it grows
How it grows
Objects are files on a handful of storage servers with three copies each, and a relational database maps keys to locations. Simple, and good enough for one product.
Objects pack into large extents, and the key index moves to range partitions that split automatically. Scrubbing and checksums become mandatory.
Sealed extents are encoded across zones, cutting raw capacity needs by almost half, and repair spreads across the whole fleet.
Cold storage tiers with slower, cheaper media join the lifecycle, the index gains strong consistency with validated caches, and regions run as independent copies of the whole system.
Watch and read
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.