Design Google Docs / Live Collaborative Editor

Picture a team of six drafting a launch plan in one document. Two of them are rewriting the same paragraph, one is fixing typos further down, and a manager is reading along with comments open. Each person sees their own typing with zero delay, because the editor applies it locally first. Within a fraction of a second, everyone else sees it too, placed correctly even though the text around it has shifted.

The system behind this has three jobs. It must order concurrent edits so all copies converge, persist every change so a crash never loses work someone saw as saved, and broadcast changes and cursors to everyone in the document with low latency. Around that core sit permissions, comments, version history, export and search, which matter for the product but are much easier to scale.

Assume 25 million people open at least one document each day, and that at the busiest hour about 4 million editor sessions are open at once. Most sessions are idle reading. Clients batch keystrokes into one message every 250 ms while the user is typing.

Document management is ordinary request and response. Editing is a long-lived, ordered stream, so it gets its own WebSocket protocol. Keep the two apart: the socket carries only what must be fast.

The design turns on one rule: every live document has exactly one owner process, the sequencer. All edits for that document pass through it, it assigns them consecutive revision numbers, and it is the only writer to that document's slice of the op log. A sequencer process hosts thousands of documents, but each document lives on one sequencer at a time.

When a client opens a document, it loads the latest snapshot and the ops after it through the document API, then opens a socket to any session gateway. The gateway asks the router which sequencer owns the document. Ownership is a lease in a small strongly consistent table; if no lease exists, the router picks a sequencer with spare capacity and the sequencer claims it. From then on the gateway forwards that client's submits to the owner and relays everything the owner broadcasts.

On each submit the sequencer transforms the incoming batch against any ops committed since the client's base revision, appends the result to the op log with a conditional write on the next revision number, then acks the author and broadcasts to the other sessions. Cursors take a separate, lossy path: they go through the same gateways but are never written down.

Positions are only meaningful relative to a particular version of the text. Every concurrent editing scheme is a way to deal with that fact. There are two families. Operational transformation keeps positions and adjusts them. CRDTs stop using positions and give every character a permanent identity instead.

A transform function takes two ops that were made against the same version and rewrites one so it can apply after the other. For two inserts, if the other insert landed at or before your position, shift yours right by its length. For an insert against a delete, shift left by however many deleted characters came before you. When two inserts land at the same position, break the tie with a fixed rule such as lower user id goes first, so every replica makes the same choice.

In the example, the server receives Ben's op first and gives it revision 101. Ana's op arrives saying "based on revision 100". The server transforms it past Ben's insert of two characters at position 0, so Ana's insert moves from position 3 to position 5, and commits it as revision 102. Ben receives insert "s" at 5 and gets a cats. Ana receives Ben's original op, transforms her own pending state the same way, and also lands on a cats.

Peer-to-peer OT is notoriously hard to get right, because transforms must satisfy properties that many published algorithms turned out to violate. The trick that makes production OT manageable is the one used by Jupiter and later by Google Wave: a single server defines the order. Each client only has to transform against one linear history, and each client has at most one unacknowledged batch in flight. The client keeps two pieces of state: the batch awaiting an ack, and a buffer of edits made since. Incoming remote ops are transformed against both before being applied to the screen.

A sequence CRDT such as RGA or YATA gives each inserted character an id made of a site id and a counter, and records which character it was inserted after. An insert says "put s after character (ana, 3)", which means the same thing on every replica no matter what else has been inserted nearby. Deletes mark a character as a tombstone instead of removing it, because a concurrent insert may still refer to it. Any replica that has received the same set of ops, in any order, renders the same text. No server needs to order anything.

The price is metadata. Every character carries an id and a parent reference, and tombstones accumulate. Modern libraries like Yjs and Automerge compress runs of characters typed in sequence, which brings the overhead down to a small multiple of the text size for typical documents, but garbage collecting tombstones safely needs to know every replica has seen the delete, which is hard when clients can be offline indefinitely. Rich text adds more subtlety: formatting spans, list structure and tables all need CRDT designs of their own.

The sequencer acks only after the op log write succeeds, so an acked edit is never lost, even if the sequencer dies the next millisecond. Writes go to a replicated store within the region, which costs a few milliseconds per batch. Batching helps: the sequencer can group submits from many sessions of the same document into one log write when traffic is heavy.

To open a document, read the newest snapshot, which records the revision it reflects, then read log rows from that revision to the head and apply them. With a snapshot every 500 ops, the tail is never long. When a sequencer takes over a document after a crash, it does the same thing to rebuild memory, then accepts submits. Clients that had a batch in flight resend it with the same client_seq; the sequencer recognises duplicates from a small per-session table stored with the log, so a retry never inserts text twice.