Design ChatGPT

Assume 40M daily users sending 8 messages each, a default model in the 70-billion-parameter class served in 8-bit weights on 8-GPU nodes, an average prompt of 2,500 tokens once history is included, and an average reply of 450 tokens.

Streaming uses server-sent events over HTTP rather than WebSockets. The traffic is one request followed by a one-way stream, SSE works through ordinary proxies and load balancers, and each event carries an index so a client can resume after a dropped connection without duplicating text.

The conversation service owns the product side. It loads the conversation branch, adds the system prompt and any memory or tool definitions, trims or summarises older turns to fit the model's context window, and saves the user's message before generation starts. It then calls the inference router with a fully built prompt and streams whatever comes back to the client.

The router decides where a request runs. It picks a model version and region, places the request in a priority queue for its tier, and chooses a replica, preferring one that already holds this conversation's earlier tokens in its prefix cache. It also enforces admission control: if every replica's KV memory is full, new requests wait in the queue rather than evicting someone mid-reply.

Inference itself splits into two kinds of work. Prefill processes the whole prompt in one parallel pass and is limited by compute. Decode generates one token at a time per sequence and is limited by how fast weights and cache can be read from GPU memory. Large deployments often run them on separate pools and hand the KV cache from one to the other. Tokens leave decode workers through a streaming moderation layer, and usage counts flow to metering on a queue.

A transformer computes, for every token, key and value vectors at every layer. Generating token 501 needs the keys and values of tokens 1 to 500. Recomputing them for each new token would make generation quadratic, so servers keep them in GPU memory as the KV cache. With the model assumed above, that is roughly 320 KB per token, or 1.3 GB for a 4K-token conversation.

Allocating that memory as one contiguous block per request wastes a lot, because you do not know in advance how long the reply will be. Paged attention, popularised by vLLM, borrows from operating systems: split the cache into fixed blocks of, say, 16 tokens, keep a block table per sequence, and allocate blocks as the sequence grows. Fragmentation drops to almost nothing, and blocks can be shared between sequences that start the same way.

The naive approach groups requests into a batch, runs it until the longest reply finishes, then starts a new batch. Short replies finish early and leave their slots idle, and new requests wait for the slowest one. Continuous batching schedules at the level of single decoding steps instead. On every step, finished sequences leave the batch and queued ones join, so the batch stays full and nobody waits for a stranger's essay to finish.

The scheduler also has to fit prefill in. A big prompt prefill can stall decoding for every other sequence on the replica for hundreds of milliseconds, which users see as a stutter. Two common fixes: chunked prefill, which splits long prompts into pieces interleaved with decode steps, and running prefill on separate workers and shipping the finished KV blocks to a decode worker over fast interconnect.

When memory runs out mid-flight, the scheduler preempts the newest or lowest-priority sequences, either moving their blocks to CPU memory or dropping them to recompute later. Doing that on purpose, under policy, beats crashing with an out-of-memory error.

Every turn of a conversation resends all previous turns, and every request shares the same long system prompt. Recomputing those tokens is wasted work. With prefix caching, KV blocks are indexed by a hash of the tokens they cover, so a new request that starts with a known prefix reuses those blocks and only prefills the new part. For a chat product, half or more of prompt tokens can come from cache, which cuts time to first token and prefill compute in step.

This only pays off if the request lands on the replica that has the blocks. The router therefore uses consistent hashing on conversation id, with a load cap, so follow-up turns usually return to the same replica. If that replica is overloaded, the request goes elsewhere and pays the full prefill; correctness never depends on the cache.

A model too large for one GPU is split. Tensor parallelism cuts each layer's matrices across the GPUs of a node; every layer needs an all-reduce between them, which is fine over the very fast links inside a node and slow across nodes. Pipeline parallelism places groups of layers on different nodes; communication is light, but each request passes through the stages in sequence. A common setup is tensor parallelism inside a node and pipeline parallelism only when the model will not fit in one node. Mixture-of-experts models add expert parallelism, routing each token to the GPUs holding its chosen experts.

Users feel two things. Time to first token depends on queueing and prefill: how long until text starts appearing. Time per output token depends on decode batch size: how smoothly the text flows. Bigger batches improve throughput and hurt per-token speed, so the scheduler caps batch size per replica to keep paid users above their tokens-per-second target. Speculative decoding helps the second number: a small draft model proposes several tokens, the large model verifies them in one pass, and accepted tokens come almost for free.