The whole design, in writing
Learn system design by building a video conferencing system like Zoom or Google Meet step by step. An interactive guide covering why real-time media needs UDP/WebRTC not TCP, why a peer-to-peer mesh explodes at N², the SFU that forwards streams and scales calls, simulcast + adaptive bitrate, signaling with SDP and ICE/STUN/TURN NAT traversal, geo-distributed and cascaded SFUs with active-speaker selection, and the unhappy paths (packet loss, congestion, recording).
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
What makes video calling hard?
A dozen faces, live, from around the world, must stay in sync with under ~150ms of delay or conversation falls apart (people talk over each other). Video is a huge, continuous stream; networks drop packets and vary wildly; and every extra participant multiplies the media. This is a real-time systems problem, not a web request.
A video conferencing system moves real-time audio/video among many participants with minimal latency, adapting to every network. The key choices: the right transport (UDP/WebRTC, not TCP), the right topology (a media server SFU, not a peer mesh), adapt quality per receiver (simulcast), and connect peers through NATs (signaling + ICE/STUN/TURN).
What the new pieces do
- Participant Aclient
- A person in the call. Captures their camera/mic and sends one uplink stream; receives a downlink stream per other participant.
Step 1 · Not a web request
Real-time media needs UDP
Your instinct might be HTTP/TCP — reliable, familiar. For live video it's exactly wrong. Why does the protocol that reliably delivers web pages ruin a video call?
Why is TCP the wrong transport for live video/audio?
Setup time matters a little, but the real killer is TCP's reliability model: it retransmits and delivers in order, so one lost packet stalls everything behind it (head-of-line blocking) — fatal for real-time media where a late frame is useless.
For live media a packet that arrives late is worthless — you want the *next* frame, not a resend of the old one. TCP's retransmit-and-wait stalls the stream; UDP lets you skip lost/late packets and keep going. WebRTC uses UDP (RTP) with its own loss handling.
TCP can carry any bytes; the issue isn't capability but semantics. Its reliable, ordered delivery is wrong for media, where timeliness beats completeness — hence UDP.
Live media values timeliness over reliability: a video frame that arrives late is useless — you want the next one. TCP's reliable, in-order delivery causes head-of-line blocking (one lost packet stalls everything behind it). So real-time media uses UDP (via RTP), tolerating loss and skipping late packets. This is what WebRTC — the browser real-time media stack — is built on.
Why this piece earns its place
Choosing UDP also means taking on everything TCP was doing for us for free — loss recovery, ordering, pacing — as our own problem, and every later step on this page is part of that bill. But the assumption worth naming now is smaller and harder: it assumes the network will pass UDP at all. Plenty of corporate, hotel and captive-portal networks block UDP outright, or allow nothing outbound except TCP on port 443, and for a user on one of those no amount of loss tolerance helps — there is no path to be timely on. So a serious client keeps a TCP-framed fallback through a relay and treats it as an explicitly degraded mode rather than an equal one, because the moment media is carried inside a reliable, ordered stream the head-of-line stall we picked UDP to escape comes straight back: a single retransmit holds up everything behind it, and it surfaces as audio drifting a beat behind the speaker’s lips. Worth knowing which path a session ended up on, too — a call that is quietly wedged on that fallback is a support ticket you can answer in one lookup instead of guessing.
Step 2 · Two people is easy
Peer-to-peer for a 1:1 call
For a simple two-person call, do you even need a server carrying the media? The lowest-latency path between two people is a straight line.
For 1:1, connect the two peers directly (peer-to-peer): each sends media straight to the other. It's the lowest latency (no server hop) and cheapest (no media server cost). This works beautifully for two people — the question is what happens when the call grows.
Why this piece earns its place
The cheap answer is right here and still loses more often than it looks. A direct connection delivers each peer’s packets to the other’s actual address, which means the two people learn each other’s IP address — fine between colleagues, a reportable privacy problem in any product that connects strangers, and reason enough that plenty of shipped products route even a two-person call through the media server. Going direct also leaves you with no vantage point on the call: no server ever touches the media, so recording, moderation, compliance capture and per-stream quality telemetry have nothing to attach to, and the only view of what happened is whatever the clients choose to report. And the two-party assumption expires the first time someone invites a third person, at which point you are changing topology during a live call — a renegotiation path that is rarely exercised and therefore rarely works. So the honest version of this step is that peer-to-peer is optimal for 1:1 on latency and cost alone, and that latency and cost are not the only things a product is buying.
Step 3 · Why the mesh explodes
N² connections
Extend peer-to-peer to a group: everyone connects directly to everyone else (a mesh). For 4 people that's fine. Why does it collapse at, say, 10 or 20?
In a full P2P mesh of N people, what blows up?
Signaling grows too, but the crippling cost is media: each participant must *upload* their camera stream separately to every other participant, so uplink bandwidth and encoding scale with N — brutal on the sender.
A mesh has no media server — that's the point. The explosion is on the *clients*: everyone sends N-1 copies of their stream and receives N-1, so upload bandwidth and CPU scale with N per person, and total connections scale as N².
In a mesh every participant encodes and uploads a separate copy of their video to each of the N-1 others, so a sender's uplink and encoding cost grow linearly with the group while total connections grow as N². Home upload bandwidth caps this at ~4-5 people.
A full mesh is O(N²) connections, and worse, each participant must upload their stream N-1 times — once to every other peer. Uplink bandwidth and encoding CPU scale with N per person, and home upload links choke at ~4–5 participants. The mesh has no server cost, but it pushes an impossible burden onto each client. Groups need a different topology.
Why this piece earns its place
That ~4–5 ceiling is not a property of the protocol — it is a property of whoever in the room has the worst upload link, so it is set by the weakest participant and changes every time the membership does. It moves with content too: a group that is comfortable at a low resolution falls apart the moment someone’s camera goes HD, because the sender’s obligation is the stream multiplied by the audience. What makes it genuinely nasty is how it fails. Nothing errors. The sender whose uplink has saturated sees a perfectly healthy call, because their own video is rendered locally and never crosses the network, while everybody else watches them freeze — so the one person who could fix it by turning off their camera is the last to find out, and every complaint about that call gets filed by somebody else, about a picture the sender has no way of seeing. Any product that ships a mesh at all therefore needs a sender-side uplink measurement and a rule that moves the call onto a server before the participants discover the ceiling for themselves.
- N²mesh connections
- N-1×uploads per person
- ~4-5mesh ceiling
What the new pieces do
- Participant Bclient
- Another participant. Renders everyone else's video and mixes their audio locally.
Step 4 · Upload once
The SFU (Selective Forwarding Unit)
The mesh's fatal flaw is that each person uploads their stream many times. What if each person uploaded once, to a server, and the server distributed it? What should that server do — and crucially, not do?
To fix the mesh, put a server in the middle. What should it do with the media?
A Selective Forwarding Unit receives each participant's single uplink stream and forwards copies to the other participants — it routes packets, it doesn't decode/re-encode. So each peer uploads once (fixing the N² sender cost) and the server stays cheap (no transcoding). This is how Zoom/Meet scale.
An MCU (mixer) does decode + composite + re-encode a single stream per participant — simple for clients but extremely CPU-heavy on the server and adds latency and inflexibility. SFUs won because forwarding is far cheaper and lets each client control its own layout.
Storing and pulling adds huge latency — fatal for a live call. The server must forward in real time, not buffer for retrieval.
Use an SFU (Selective Forwarding Unit). Each participant uploads one stream to the SFU; the SFU forwards (routes, without decoding/re-encoding) that stream to every other participant. Now each peer uploads once — killing the mesh's N² sender cost — and the SFU stays cheap because it only forwards packets, not transcodes. This is the topology behind modern large calls.
Why this piece earns its place
Forwarding is what makes the SFU cheap, and it is equally the constraint we are accepting: the server never decodes, so it can never improve on what a sender gave it. It cannot shrink a stream for a struggling receiver, it cannot re-encode for an endpoint that speaks a different codec, and it cannot compose a picture — which is why the quality problem has to be handed back to the sender rather than solved on the box, and why anything that genuinely needs pixels has to subscribe like a participant and pay for its own decode. The other half of the cost did not vanish either; it moved. Each peer uploads once now, but the SFU still sends every stream to every other participant, so traffic across the server’s network interface grows with the square of the room while no individual client feels any of it. That makes capacity planning a count of forwarded streams, not of calls, and since many calls share a machine, one very large meeting becomes a noisy neighbour for every unrelated call on it. Admission control — knowing how many streams a box will accept before it accepts them — matters more here than raw bandwidth does.
- upload 1×per participant
- forwardno transcode
- SFUscales the call
What the new pieces do
- SFUservice
- Selective Forwarding Unit: each peer uploads one stream; the SFU forwards it to every other peer. The scalable hub of the call.
Step 5 · Everyone's network differs
Simulcast & adaptive bitrate
One participant is on gigabit fiber, another on a spotty phone. If the SFU forwards the same high-quality stream to both, the phone user's video freezes and stutters. But the sender uploads only one stream — how can each receiver get a quality that fits their network?
The sender uploads one stream, but receivers have wildly different bandwidth. How does each get a fitting quality?
A weak receiver can't cope — its video freezes and audio breaks. You must send each receiver a quality matched to its bandwidth without burdening the sender.
That's the mesh problem again — sender uplink scales with N. The trick is to upload a few quality layers once and let the SFU pick per receiver.
With simulcast (or SVC) the sender encodes and uploads a few resolutions/bitrates simultaneously; the SFU, watching each receiver's reported bandwidth, forwards the highest layer that fits. One upload effort, per-receiver quality — the phone gets low-res, the fiber gets HD, and the SFU adapts in real time.
Use simulcast (or scalable video coding, SVC): the sender uploads a few quality layers at once (e.g. low/medium/high). The SFU, tracking each receiver's available bandwidth (via congestion feedback), forwards the highest layer that fits each receiver — HD to the fiber user, low-res to the phone. And it can drop to audio-only when needed. One upload, adaptive per-receiver quality, all decided at the forwarding server.
Why this piece earns its place
Simulcast is not free for the sender: encoding several versions of one camera costs real CPU and more total uplink than encoding one, and it lands hardest on exactly the device most likely to be constrained — a phone that is already thermally throttling. So the layer count is a knob rather than a constant. Drop to two layers, or to SVC where a single layered stream can be thinned instead of separately re-encoded, whenever the sender is the constrained party rather than the receiver. The second knob is how eagerly the SFU switches. Promoting a receiver to a higher layer requires a keyframe from that layer, and a keyframe is a large frame, so every optimistic upgrade puts a burst on the precise link you were unsure about. Follow the bandwidth estimate too closely and a receiver oscillates between resolutions, which people find more distracting than simply sitting on the lower one. The setting that works is deliberately asymmetric: switch down immediately, because the alternative to a blurry picture is a frozen one, and switch up slowly, only once the headroom has held for a while.
- 3 layerslow / med / high
- per receiverbest-fit layer
- adaptivefollows congestion
What the new pieces do
- Participant Cclient
- A participant on poor bandwidth — the reason we send multiple quality layers and adapt per receiver.
Step 6 · Just connecting is hard
Signaling & NAT traversal
Before any media flows, two peers on the internet — both behind home routers (NAT) with no public address — have to find a network path to each other and agree on codecs. Neither can just be "dialed." How do they connect at all?
Use a signaling server to exchange setup info: SDP offer/answer (codecs, parameters) and ICE candidates (possible network paths). To pierce NAT: STUN tells a peer its own public address so peers can try a direct connection; when a direct path is impossible (strict/symmetric NAT, firewalls), TURN relays the media through a server as a fallback. Signaling only sets up the call; once connected, media flows peer↔SFU directly.
Why this piece earns its place
Two details here decide how this feels in production. The first is that ICE is the part users experience as joining. If a client gathers every candidate — local, STUN-discovered, relay — before it sends its offer, the slowest candidate sets the join time for everyone. Trickle ICE sends candidates as they are found and starts checking pairs immediately, so a direct path is usually live before the relay candidate has even been allocated. The same machinery has to run again mid-call: a phone crossing from wifi to cellular changes address, and without an ICE restart the media just stops on a connection that still believes it is healthy. The second is that signaling is a separate availability domain from media, and that is easy to misread. If signaling dies, every call already in progress keeps running — media goes peer to SFU and never touches it — so the traffic graphs stay calm while nobody can join, nobody can renegotiate, and nobody who drops can get back in. That is an outage hiding behind healthy numbers, which is why it needs its own alarm on join success rather than on concurrent streams.
- SDPcodecs / params
- STUNdiscover public IP
- TURNrelay when direct fails
What the new pieces do
- Signalinggateway
- Sets up connections: exchanges SDP offers/answers and ICE candidates so peers can find a path to each other.
- STUN / TURNservice
- STUN discovers a peer's public address; TURN relays media when a direct connection is impossible (strict NAT/firewall).
Step 7 · Big and global calls
Geo-distributed & cascaded SFUs
A single SFU has limits — its bandwidth caps how many streams it can forward, and a call with participants on different continents shouldn't route everyone through one distant server. How do you scale to huge, global calls?
Distribute SFUs geographically so each participant connects to a nearby one (low latency), and cascade them: SFUs peer with each other so a stream crosses between regions once rather than every remote participant pulling it across the ocean. For very large calls (webinars, thousands of viewers), forward only what matters — active-speaker selection and a few top video streams — instead of everyone's stream to everyone. The topology becomes a tree of SFUs, not one hub.
Why this piece earns its place
Cascading buys locality and pays for it in hops. Each additional SFU between a speaker and a listener is another queue inside the ~150ms budget, so the thing to bound is hops per stream rather than servers per region: one hop between any two regions is a design, and a chain of relays that accumulated as regions were added is a slow outage nobody declared. The second thing this step quietly introduces is editorial. Once we stop forwarding everyone’s video, the system is deciding what each person sees, and it fails in a very specific way — a cough, a closing door, a laugh flips the active speaker and the grid churns while the actual conversation is elsewhere. So the switch needs a hold time and a margin over the runner-up rather than a bare loudest-wins comparison, and ideally a pinned position for whoever is presenting. Audio should stay out of that selection entirely: audio streams are small, and a participant muted by the layout because they were off screen cannot interrupt — which is the one affordance a live conversation genuinely cannot do without.
- geo SFUsconnect to nearest
- cascadecross regions once
- active speakerforward what matters
What the new pieces do
- Cascade SFUservice
- A second SFU (another region / large call) that peers with the first, so big and global calls fan out without one server carrying everyone.
Step 8 · The sharp edges
Loss, congestion, echo & recording
Real networks are hostile: packets drop, bandwidth fluctuates, audio echoes, and you often need to record the call — all while keeping latency low.
Survive packet loss with a jitter buffer (smooth timing), NACK (selective retransmit of just-lost packets when time allows), and FEC (forward error correction — send redundancy so small losses self-heal). Run congestion control (estimate available bandwidth and tell simulcast which layer to send) so the call adapts instead of collapsing. Do echo cancellation / noise suppression on the client audio. And record by attaching a special subscriber that mixes and transcodes the streams to storage — off the live path.
Why this piece earns its place
NACK and FEC look like two tools for the same job, but they spend different currencies and the choice between them is mostly about round-trip time. A retransmit costs nothing until something is actually lost, yet it only helps if the replacement lands before the jitter buffer has to play that instant — on a long path it is simply too late to ask. Redundancy costs bandwidth on every packet whether or not anything is lost, and in exchange repairs small gaps with no round trip at all. The split falls out of that: audio is cheap to protect and the most damaging thing to lose, so it can carry redundancy continuously, while video is the expensive stream and leans on retransmission while there is still time, accepting a dropped frame when there isn’t. The trap is that recovery traffic is still traffic. Redundancy and retransmits are counted by the same bandwidth estimate that drives which simulcast layer goes out, so a client that responds to loss by adding more repair pushes the link further into congestion, loses more, and repairs harder. Loss recovery has to sit inside the congestion controller’s budget, not alongside it, or the healing mechanism becomes the thing breaking the call.
What the new pieces do
- Recordingdata
- Subscribes like a participant to mix/record the call and transcode it for later playback.
You did it
You just designed a video conferencing system.
Everything you assembled, in order
- Real-time media values timeliness over reliability — use UDP/WebRTC, drop late packets, not TCP.
- A direct peer-to-peer connection is ideal for a 1:1 call (lowest latency, no server).
- A full mesh is O(N²) and forces each peer to upload their stream N-1 times — it collapses at ~5 people.
- An SFU lets each peer upload once and forwards streams (no transcoding) — the scalable topology.
- Simulcast + adaptive bitrate let the SFU send each receiver a quality that fits its network.
- Signaling (SDP) + ICE/STUN/TURN connect peers across NATs; TURN relays when direct fails.
- Geo-distributed, cascaded SFUs + active-speaker selection scale to huge, global calls; loss/congestion handled with jitter buffers, NACK, FEC.