Infrastructure
Design Code Deployment Pipeline
01
What the pipeline must do
What the pipeline must do
- On every merge to main, build each affected service, run its tests and publish an immutable artifact.
- Engineers deploy an artifact to staging or production by naming its digest, or let a service deploy automatically after merge.
- Production rollouts go in waves: a canary slice, then a region, then the rest, with health checks between each.
- A rollout pauses or rolls back automatically when error rate, latency or saturation gets worse than the baseline.
- Anyone can see what version runs where, who shipped it, and what changed since the last one.
- Teams can freeze deploys for a service or the whole company, with an audited override.
- Builds are reproducible: the same source and toolchain give a byte-identical artifact.
- Only artifacts built by the trusted pipeline, with signed provenance, can run in production.
- Median merge to production under 30 minutes for a typical service.
- Rollback to the previous version completes within 3 minutes for any service.
- The pipeline itself never becomes the reason nobody can ship a fix during an incident.
Ask whether the company runs a monorepo or many repos, whether services run on Kubernetes, VMs or both, and whether database migrations ship with code. Those three answers decide how you detect what to build, what a deploy actually does on a host, and whether rollback is always safe. Also ask who the users are: a pipeline for 40 engineers and one for 4,000 are different products.
02
Sizing the pipeline
Sizing the pipeline
Assume 4,000 engineers working on 800 services across six regions. About 6,000 changes merge each day, and each pull request is also built several times before it merges. Production runs about 120,000 containers.
| Quantity | Working | Result |
|---|---|---|
| Builds per day | 6,000 merges + about 19,000 pull request builds | about 25,000 |
| Build minutes per day | 25,000 x 12 min average | 300,000 minutes |
| Average concurrent builds | 300,000 / 1,440 | about 210 |
| Peak concurrent builds | 3x average in the working-hours overlap | about 630 workers of 8 vCPU |
| Production deploys per day | assumption, many services auto-deploy | about 3,000 |
| New artifact data per build | container image of about 400 MB, about 60 MB in new layers | about 1.5 TB a day |
| Artifact storage | 30 days of every build plus every released version forever | about 60 TB and growing slowly |
| Bytes pulled for deploys | 3,000 deploys x 150 hosts x 60 MB of new layers | about 27 TB a day |
| Canary metrics | 3,000 deploys x 20 metrics x 2 groups, sampled every 10 s | light, but every query must be fast |
Build compute dominates cost and artifact pulls dominate network. Both are cut sharply by caching: a remote build cache skips work whose inputs have not changed, and regional registry mirrors keep 27 TB a day of pulls off the central store.
03
API design
API design
04
Architecture overview
Architecture overview
flowchart LR
G["Git: merge to main"] --> CI["CI orchestrator"]
CI --> Q[["Build queue"]]
Q --> W["Ephemeral build workers"]
W --> RC[("Remote build cache")]
W --> AR[("Artifact registry, by digest")]
W --> PV["Provenance signer"]
PV --> AR
AR --> MR[("Regional mirrors")]
CI --> DS["Deploy service: desired versions"]
DS --> RO["Rollout controller"]
RO --> CL["Clusters in 6 regions"]
CL --> MR
CL --> MT[("Metrics store")]
MT --> CJ["Canary judge"]
CJ --> ROThe pipeline has two halves joined by one thing: an artifact digest. The build half turns a commit into a signed artifact whose name is the SHA-256 of its contents. The deploy half takes a digest and moves it through environments. Nothing in the deploy half ever rebuilds, so what was tested is exactly what ships.
Build workers are ephemeral. Each job gets a fresh sandbox with no network access except to the cache and a pinned dependency mirror, runs the build and is destroyed. Before running any step, the worker hashes that step's inputs and asks the remote cache whether the output already exists. In a large monorepo, 80% or more of steps are cache hits, which is what keeps a 12-minute average from becoming an hour.
The deploy service stores desired state: for each service and environment, which digest should run and with what rollout strategy. The rollout controller compares desired with actual, moves traffic in waves and asks the canary judge after each wave whether to continue. Admission control in every cluster refuses any image without a valid signature from the pipeline's provenance signer.
Finds the targets affected by a commit from the build graph, schedules them and records results. Prioritises merge builds over pull request builds when the queue is long.
Content addressed and immutable. Tags are human labels pointing at digests; deploys always use the digest, never a tag such as latest.
A reconcile loop per rollout: shift a slice of traffic, wait, ask the judge, continue or reverse. Survives its own restart because state lives in the deploy service.
Compares canary pods with baseline pods started at the same time on the old version, metric by metric, and returns pass, fail or inconclusive.
05
Deep dive: builds you trust, rollouts that judge themselves
Deep dive: builds you trust, rollouts that judge themselves
Builds you can reproduce, artifacts you can trust, and rollouts that notice trouble before a human does. Each is a separate mechanism.
Hermetic, cacheable builds
A build is hermetic when its output depends only on declared inputs: source files, pinned dependency versions by hash, and a pinned toolchain image. No reading the clock, the network or the host's installed packages. Tools such as Bazel or Buck enforce this by running each action in a sandbox that only sees its declared inputs.
Hermeticity is what makes caching safe. The cache key for an action is the hash of its command line and every input file. If two engineers, or two CI runs, hit the same key, the second just downloads the output. Without hermeticity, a cache hit might hide a difference in environment, and you end up with an artifact that works only because of what happened to be on one worker. That is the classic "works in CI, breaks in prod" story.
Trusting the artifact
When a build finishes, the signer writes a provenance statement: which commit, which build definition, which builder, which input digests. It signs that statement with a key only the pipeline holds and stores it next to the artifact. This follows the shape of the SLSA framework. Clusters run an admission check that verifies the signature and refuses anything that did not come from the pipeline, so a laptop-built image or a tampered registry entry cannot run in production.
Pin third-party dependencies by hash in a lockfile and pull them through an internal mirror. When a public package is compromised, you can answer "which of our artifacts contain it" from provenance data in minutes instead of searching every repo.
Waves and the canary judge
flowchart TD
ST["Rollout starts with digest D"] --> C1["Wave 1: 1% in one region, plus matching baseline"]
C1 --> J1{"Judge after 10 minutes"}
J1 -->|fail| RB["Roll back every wave"]
J1 -->|inconclusive| EX["Extend wave, collect more data"]
EX --> J1
J1 -->|pass| C2["Wave 2: 100% of first region"]
C2 --> J2{"Judge after 15 minutes"}
J2 -->|fail| RB
J2 -->|pass| C3["Wave 3: remaining regions, two at a time"]
C3 --> J3{"Final judge and bake time"}
J3 -->|pass| DONE["Mark D as last good"]
J3 -->|fail| RBComparing a canary against the whole fleet is misleading, because new pods have cold caches and fresh JIT state. Start a small baseline group on the old version at the same moment as the canary, and route equal traffic to both. Now the only difference between the groups is the code.
For each metric, such as error rate, p50 and p99 latency, CPU per request and memory growth, the judge runs a statistical test, for example Mann-Whitney on latency samples, and classifies the difference as better, same or worse beyond a threshold. A rollout fails if any critical metric is worse, and is inconclusive when there is too little traffic to tell. Low-traffic services get longer waves rather than weaker tests.
Bake time matters for slow failures. Memory leaks, cache poisoning and daily batch jobs do not show up in ten minutes. The final wave waits for a bake period before the digest is marked last good, the target every future rollback uses.
Coordinating 3,000 rollouts a day
Each rollout is a record in the deploy service with a state: pending, canary, wave_n, baking, done, rolling_back or rolled_back. Only one active rollout per service and environment is allowed, enforced by a conditional write on that pair. A new digest arriving while one is in flight either queues behind it or, if the author asks, supersedes it, in which case the controller finishes the current wave and switches target.
Rollouts also interact. If a shared dependency such as the auth service is mid-rollout and a downstream service starts failing, the judge may blame the wrong change. Record every active rollout per region with start times, and when a judge fails a wave, show the other rollouts that overlapped it. Company-wide freezes are just a flag the deploy service checks before admitting any new rollout, with an override that needs a second approver and is logged.
Every state change is an event. The what-runs-where page, Slack notifications and the audit log are all projections of that stream, so they never disagree with the controller.
Separating deploy from release
Feature flags let code reach production switched off and then be enabled for 1%, 10% and 100% of users independently of the deploy. That shrinks what a deploy can break: a bad deploy is mostly a crash or performance problem, which canaries catch well, while a bad feature is turned off with a flag in seconds. Flags need hygiene, an owner and an expiry date each, or the codebase fills with dead branches.
If version N+1 runs a migration that drops a column version N reads, rolling back breaks everything. Use expand and contract: add the new column first, ship code that writes both, backfill, move reads, and only drop the old column in a later release once no running version needs it.
06
Designs that look fine and are not
Designs that look fine and are not
A deploy references checkout:latest. Between staging and production someone pushes a new image under the same tag, and production gets code that was never tested.
Staging and production each run their own build from the same commit. A changed base image or dependency between the two builds produces different binaries.
Build once, promote the same artifact through every environment, and inject environment differences as configuration.A bad change reaches all users within a minute and there is no healthy region to compare with or fail over to.
Roll out in waves with a small canary first and region boundaries as natural blast-radius limits.At 3,000 deploys a day nobody can watch them all, and a tired engineer misses a 30% latency rise at 2 a.m.
Automate canary analysis with a baseline, and make rollback the default action when the judge fails a wave.State leaks between builds, a compromised job can poison later ones, and "clean the workspace" becomes a ritual.
Run every build in a fresh sandbox and destroy it afterwards; rely on the remote cache, not local leftovers, for speed.07
Decisions and their costs
Decisions and their costs
Reproducible outputs, safe caching and big speed-ups in a monorepo. Costs real migration effort: every build rule must declare its inputs, and some tools fight sandboxing.
Pays back at this sizeFast to set up and familiar. Caching is coarse and occasionally wrong, and reproducibility depends on discipline. Fine for a few dozen services.
Good for small teamsCatches regressions at about 1% of traffic, with few false alarms because canary and baseline share conditions. Needs good metrics per pod and enough traffic.
Automatic and trustworthyTwo full environments and one traffic flip. Rollback is instant, but the cost is double capacity during deploys and every user moves at once.
Useful for stateful or tiny servicesThe controller drives each wave and can react within seconds. It is a central component that must be highly available.
Fast reactionClusters reconcile from a Git repo of desired state, giving a clean audit trail and simple disaster recovery. Wave orchestration and fast rollback need extra tooling on top.
Strong for config and audit08
Failure modes
Failure modes
A change to a shared library invalidates the cache for 600 services at once just as everyone merges before a holiday.
Prioritise merge builds and hotfixes over pull request builds, autoscale workers from queue depth, and warn authors before merging changes with a huge rebuild footprint.The regression only affects one rare request type, which the canary saw too few times to judge.
Weight metrics by endpoint, extend waves for low-traffic services, add synthetic probes for critical paths and keep bake time before marking last good. Track escaped regressions and tune thresholds from them.A storage fault serves a layer whose bytes do not match its digest. Hosts fail to start the new version.
Clients verify digests on pull and retry from another mirror. Because artifacts are content addressed, corruption is detectable by construction and never runs silently.Version N+1 wrote data in a new format that version N cannot read. Rolling back turns a latency problem into errors.
Require backward-compatible data and schema changes, test rollback as part of the canary wave, and block rollouts whose migrations are not reversible without an explicit override.A popular open source package ships a malicious release, and it is pulled into builds within hours.
Pin dependencies by hash, pull only through a mirror that delays brand-new versions, scan in the pipeline, and use provenance to list affected artifacts and block their deploy at admission.The CI orchestrator's database fails while production is on fire, and nobody can ship the fix.
Run the deploy path independently of CI, keep a break-glass procedure that deploys a known signed digest directly, and drill it regularly.09
How to handle the interview
How to handle the interview
Split build from deploy. Say "build once, deploy the same digest everywhere" early. It is the backbone of the design.
Explain why hermetic matters. Tie it to caching and reproducibility. Without declared inputs, a cache hit can be a lie.
Describe the canary with a baseline. Most candidates compare canaries with the whole fleet. A same-age baseline and a statistical test stand out.
Make rollback automatic. At thousands of deploys a day, a failed judgement should reverse the rollout without paging anyone.
Mention schema compatibility. Expand and contract migrations show you know rollback has limits.
Cover supply chain security. Signed provenance and admission control are now standard expectations, not extras.
Plan for the pipeline failing. A break-glass path proves you thought about the worst day, not the average one.
10
Growing the pipeline in stages
Growing the pipeline in stages
Every step below removes a pain you can point to. Do not build a canary judge before you have per-pod metrics worth judging.
Build a container on merge, push it to a registry, run a rolling update. Rollback is redeploying the previous tag.
Deploy by digest, promote the same artifact from staging to production, and add a deploy service that records what runs where.
Rollouts move in waves with a baseline comparison, and failed judgements reverse automatically. The human dashboard check is no longer the safety net.
Build graphs decide what to rebuild, caches cut build time, and admission control enforces that only pipeline-built artifacts run.
Most services deploy every merge on their own, releases are controlled with flags, and humans step in only when the judge says inconclusive.
Watch and read
Further watching and reading
Try next
Free to read · better with landed
Whiteboard this with landed
landed runs it as a live system design round on the whiteboard and grades your trade-offs.