Design Video Conferencing

Picture a weekly team meeting: nine people, three countries, two on laptops in offices, one on a phone in a car, and one sharing a slide deck. Each of them sends one camera and one microphone, and each wants to see and hear everyone else with no noticeable delay. When the person in the car drives under a bridge, everyone else should notice nothing at all.

The naive design, every client sending its stream to every other client, needs each laptop to upload eight copies of its video. Mixing all the streams on a server into one picture fixes upload but means decoding and re-encoding every stream for every meeting, which is ruinously expensive and adds delay. The design that wins in practice sits between them: a server that forwards packets without decoding them, and lets each viewer receive only what its screen and network can use.

Assume 25 million people in calls at the daily peak, an average meeting of six participants, and a typical receive budget of four visible videos plus audio. About 15% of participants are behind networks that block UDP and must be relayed over TCP.

Think of the system as three planes with very different properties. The control plane is ordinary web software: accounts, scheduling, the meetings database and the placement service that decides which media server hosts a meeting. It can tolerate a few hundred milliseconds of latency and is deployed in a handful of cloud regions.

The signalling plane keeps one WebSocket per participant. It tells clients who is in the room, exchanges the session descriptions that set up media, and carries host actions, chat and reactions. Room state, such as the participant list and who is muted, lives in a small replicated store keyed by meeting ID, so any signalling node can serve any room.

The media plane is where all the bandwidth goes. It is a fleet of selective forwarding units at about 60 edge locations, close to where people are. Each participant connects to its nearest edge. If all participants share an edge, one SFU forwards everything; if they are spread across continents, the edges they use are cascaded over a private backbone, so each stream crosses an ocean once rather than once per viewer.

Production systems mix them. A 1:1 call starts peer to peer for the lowest latency and switches to an SFU when a third person joins or when the direct path fails. Phone dial-in goes through a small audio mixer, because a phone line can only carry one mixed audio stream.

Each sender encodes its camera three times: a high layer at 720p, a medium layer at 360p and a low layer at 180p. The SFU receives all three and sends each viewer the one that fits. A viewer with the speaker in a large tile gets the high layer; a viewer showing that person as a thumbnail gets the low layer; a viewer whose network is struggling gets whatever fits under its estimated bandwidth.

Switching layers cleanly needs care. The SFU can only switch a viewer to a new layer at a keyframe, so it asks the sender for one with a picture loss indication and switches when it arrives. It also rewrites sequence numbers and timestamps so the receiver sees one continuous stream. With codecs that support scalable video coding, such as VP9 and AV1, the layers are nested inside one stream and the SFU can drop the upper layers without waiting for a keyframe.

Every receiver runs congestion control, typically Google's GCC or a transport-wide feedback variant, which estimates available bandwidth from packet arrival times and loss. The SFU reads that estimate and walks a ladder, from the richest experience down to the minimum that still works:

Most clients sit behind NAT and cannot receive inbound packets from a server they have not contacted. WebRTC solves this with ICE: the client gathers candidate addresses, including its public address learned from a STUN server, and tests which ones reach the SFU. Because the SFU has a public address, a direct UDP path works for about 85% of participants. The rest are behind corporate firewalls or strict NATs and fall back to TURN over TLS on port 443, which looks like ordinary HTTPS traffic. That path adds 10 to 30 ms and suffers when packets are lost, because TCP retransmits audio that is already too late to play.

Packets on the internet arrive late, out of order or not at all. The receiver keeps a jitter buffer that holds packets for 20 to 80 ms and adapts its size to measured jitter: bigger means smoother playback but more delay. For lost video packets the receiver sends a NACK and the SFU, which keeps a few hundred milliseconds of recent packets, retransmits from its own buffer; this is fast because the SFU is only one short hop away. Audio uses Opus with in-band forward error correction, and when a packet is truly gone, packet loss concealment synthesises a few milliseconds of plausible sound.

If a meeting has people in Mumbai, Frankfurt and São Paulo, forcing everyone onto one SFU makes two of the three groups cross an ocean on the public internet. Instead, each group connects to its local edge, and edges exchange streams over a private backbone. An edge forwards each remote sender's stream once to each other edge, whatever the number of viewers there. That keeps every participant's first hop short, routes long distances over a managed network, and turns a meeting into a small tree of servers.

By default every stream is encrypted with DTLS-SRTP between the client and the SFU. The SFU can read headers it needs for forwarding, and it could decrypt the payload, which is what makes server-side recording and live captions possible. With end-to-end encryption turned on, clients add a second layer of encryption to each frame using keys the server never sees, through insertable streams and a scheme such as SFrame. The SFU still forwards packets and chooses layers from unencrypted metadata, but recording, transcription and phone dial-in stop working, which is why it is usually an opt-in setting.

A recording worker joins the meeting as a hidden participant on the same SFU and subscribes to everything. It writes the raw media to durable storage in short segments first, so a crash loses at most a few seconds, then composes them into one layout and encodes the final file after the meeting. Keeping composition off the live path means a busy recording cluster can never make a call worse.