Design Collaborative Whiteboard

Assume 25 million monthly users, 3 million of them active on a given day, with sessions that average 35 minutes. Most boards are small, but the design is judged on the heavy ones: workshop boards that collect tens of thousands of objects over a year.

Live collaboration runs over a single WebSocket per open board. Plain HTTP handles everything that is not real time: creating boards, uploading images, history and exports.

Each open board lives on exactly one room server at a time. The board router hashes board_id onto the room fleet and pins every socket for that board to the same process, so the server can merge edits in memory and fan them out without asking anyone else. A room with nobody connected costs nothing: the room server loads the board when the first user arrives and drops it a few minutes after the last one leaves.

An edit takes two paths at once. The room server applies it to its in-memory copy, broadcasts it to the other sockets in the room, and appends it to the board's stream in the op log. Only after the append succeeds does it send an ack to the author. The client keeps unacknowledged operations in a local queue, so a crash between broadcast and append loses nothing: the author simply resends them to whichever server takes over.

Behind the live path, a compactor folds the op log into tiled snapshots every few hundred operations or every minute, whichever comes first. Snapshots are what cold loads, history views and exports read, which keeps those workloads off the room servers.

A whiteboard is not a text document. There is no single sequence that everyone inserts into. It is a set of independent objects, each with a handful of properties: position, size, colour, text, z-order, parent frame. That shape suits a CRDT very well. The board is a map from object id to object, and every object is a map from field name to a last-writer-wins register.

Each register stores its value plus a stamp of (lamport_clock, client_id). When two updates to the same field meet, the higher clock wins, and the client id breaks ties. The rule is deterministic, so every replica picks the same winner no matter which update it saw first. Two people dragging the same note at once is the common conflict; with this model one of them ends up holding it, and nobody sees the note split in two.

Field granularity is what makes this pleasant. If Priya recolours a note while Marco moves it, those are different registers and both changes survive. Only a true clash on the same field resolves by stamp. Text inside a note or a text box is the exception: two people typing in one paragraph need a sequence CRDT for that field, the same kind a document editor uses, so letters interleave instead of one person's sentence replacing the other's.

Creation and deletion fit the same scheme. Creating an object is writing a new key with a client-generated id, so no server round trip is needed to get an id. Deleting sets a deleted register to true. The object becomes a tombstone that sticks around until compaction can prove every replica has seen it, which is what stops a late "move" from resurrecting a note someone already threw away.

Stacking order is a classic trap. Storing an integer index means "bring to front" renumbers many objects, and concurrent reorders collide. Instead give each object a fractional index, a string key that sorts lexically. Putting a shape between keys "a3" and "a4" generates something like "a3V". One register changes per reorder, and if two people pick the same key the client id breaks the tie.

The room server keeps every object's bounding box in a spatial index. An R-tree works; so does a simple quadtree of fixed tiles, which has the advantage that tiles double as cache keys for cold loads. When a client says its view is the rectangle [-1200, 300, 2400, 1400], the server finds the tiles that overlap it, plus a margin of one tile on every side so that a small pan does not trigger a fetch, and streams only those objects.

Zoom changes the picture. At 5% zoom a viewport might cover the whole 60,000-object board, and drawing every sticky note is pointless when each is two pixels wide. So each tile also stores a level-of-detail summary: coloured rectangles for clusters of notes and frame titles, generated by the compactor. Below a zoom threshold the client renders summaries; it fetches real objects only once the user zooms in far enough to read.

Edits respect the same interest set. When someone moves a note, the room server sends the update to sockets whose padded viewport contains the note's old or new box. Others receive nothing until they pan there, at which point the tile they request already includes the change. For a 300-person all-hands board where everyone is zoomed into different corners, this cuts fan-out by an order of magnitude.

A shared undo stack is wrong for multiplayer. If I press Ctrl+Z, I expect my last move to revert, not the note my colleague just wrote. So every client keeps its own undo stack, and each entry records the inverse of a local action: for a move, the previous position; for a delete, the full object as it was.

Undo never rewinds history. It issues a new forward operation that writes the old values back, stamped with a fresh clock, so it merges like any other edit and every replica agrees on the result. Redo is the same trick in the other direction.

The subtle case is when someone else touched the same field after you. Say you moved a note left, then a teammate moved it right. If your undo blindly restored your old position, it would trample their newer move. A good rule: before applying an inverse, compare the field's current stamp with the stamp your action produced. If someone else has written since, skip that field (or the whole entry) and move on. Undo should feel like it rewinds your intent, and quietly giving up on a contested field is less surprising than reverting a teammate's work.

Group operations into undo entries by gesture, not by message. A two-second drag might send forty position updates, but one Ctrl+Z should put the note back where the drag started. The client opens an entry on pointer-down and closes it on pointer-up.

Cursors are the noisiest data on the board and the least valuable to keep. Clients sample the pointer at most every 50 ms and skip a send if it moved less than a couple of pixels. The presence relay merges all cursors in a room into one frame per recipient every 100 ms, includes only people whose cursor falls inside that recipient's padded viewport, and caps a frame at the 25 nearest cursors. Receivers interpolate between frames so motion still looks smooth. Losing a frame is harmless, which is why presence uses no acks and no log.

Offline edits are just operations that waited. The client writes them to IndexedDB as it goes. On reconnect it sends its state vector, the highest clock it has seen from each client id, and the server answers with exactly the operations the client is missing; the client then sends its queue. Because the CRDT merges in any order, no special conflict screen is needed. If the client has been away so long that the server has compacted past its state vector, the server sends fresh tiles for the viewport instead, and the client replays its queue on top.