The big idea
What makes video calling hard?
A dozen faces, live, from around the world, must stay in sync with under ~150ms of delay or conversation falls apart (people talk over each other). Video is a huge, continuous stream; networks drop packets and vary wildly; and every extra participant multiplies the media. This is a real-time systems problem, not a web request.
A video conferencing system moves real-time audio/video among many participants with minimal latency, adapting to every network. The key choices: the right transport (UDP/WebRTC, not TCP), the right topology (a media server SFU, not a peer mesh), adapt quality per receiver (simulcast), and connect peers through NATs (signaling + ICE/STUN/TURN).
How to read this: Each step opens with a real design decision — make the call before I show you what ships. Watch the topology grow, and at the end kill the SFU and kill TURN to see the cost of centralizing and the role of relays. Hit Begin.
Step 1 · Not a web request
Real-time media needs UDP
Your instinct might be HTTP/TCP — reliable, familiar. For live video it's exactly wrong. Why does the protocol that reliably delivers web pages ruin a video call?
Design decision: Why is TCP the wrong transport for live video/audio?
The call: TCP guarantees in-order, reliable delivery — so a lost packet stalls the stream (head-of-line blocking); real-time media prefers UDP and drops late data. — For live media a packet that arrives late is worthless — you want the *next* frame, not a resend of the old one. TCP's retransmit-and-wait stalls the stream; UDP lets you skip lost/late packets and keep going. WebRTC uses UDP (RTP) with its own loss handling.
Live media values timeliness over reliability: a video frame that arrives late is useless — you want the next one. TCP's reliable, in-order delivery causes head-of-line blocking (one lost packet stalls everything behind it). So real-time media uses UDP (via RTP), tolerating loss and skipping late packets. This is what WebRTC — the browser real-time media stack — is built on.
WebRTC + UDP/RTP: WebRTC gives browsers real-time media over UDP: RTP for the media, plus built-in encryption, congestion control, and loss recovery (NACK/FEC). The mental shift from web dev: you're not fetching a resource reliably, you're streaming perishable data where "old" means "drop it."
Step 2 · Two people is easy
Peer-to-peer for a 1:1 call
For a simple two-person call, do you even need a server carrying the media? The lowest-latency path between two people is a straight line.
For 1:1, connect the two peers directly (peer-to-peer): each sends media straight to the other. It's the lowest latency (no server hop) and cheapest (no media server cost). This works beautifully for two people — the question is what happens when the call grows.
P2P is ideal — for two: A direct peer connection is optimal for 1:1: minimal latency, no server bandwidth. The entire scaling story of video conferencing is about what to do when "just connect the peers" stops working — which happens fast as you add people.
Step 3 · Why the mesh explodes
N² connections
Extend peer-to-peer to a group: everyone connects directly to everyone else (a mesh). For 4 people that's fine. Why does it collapse at, say, 10 or 20?
Design decision: In a full P2P mesh of N people, what blows up?
The call: Each peer must send its stream to all N-1 others — uplink/CPU scale with N per person; total links ~N². — In a mesh every participant encodes and uploads a separate copy of their video to each of the N-1 others, so a sender's uplink and encoding cost grow linearly with the group while total connections grow as N². Home upload bandwidth caps this at ~4-5 people.
A full mesh is O(N²) connections, and worse, each participant must upload their stream N-1 times — once to every other peer. Uplink bandwidth and encoding CPU scale with N per person, and home upload links choke at ~4–5 participants. The mesh has no server cost, but it pushes an impossible burden onto each client. Groups need a different topology.
The N² wall: Mesh is peer-to-peer's downfall at scale: the sender-side cost (upload the same video to everyone) is the real killer, not just the connection count. To scale, each peer should upload their stream once — which means a server in the middle.
Step 4 · Upload once
The SFU (Selective Forwarding Unit)
The mesh's fatal flaw is that each person uploads their stream many times. What if each person uploaded once, to a server, and the server distributed it? What should that server do — and crucially, not do?
Design decision: To fix the mesh, put a server in the middle. What should it do with the media?
The call: Decode all streams, mix them into one video, and send that (an MCU). — An MCU (mixer) does decode + composite + re-encode a single stream per participant — simple for clients but extremely CPU-heavy on the server and adds latency and inflexibility. SFUs won because forwarding is far cheaper and lets each client control its own layout.
Use an SFU (Selective Forwarding Unit). Each participant uploads one stream to the SFU; the SFU forwards (routes, without decoding/re-encoding) that stream to every other participant. Now each peer uploads once — killing the mesh's N² sender cost — and the SFU stays cheap because it only forwards packets, not transcodes. This is the topology behind modern large calls.
SFU vs MCU: An SFU forwards streams (cheap server, each client gets separate streams and controls its own layout). An MCU decodes and mixes everyone into one composite stream (simple clients, but heavy server CPU and less flexible). SFUs dominate because forwarding scales far better than mixing.
Step 5 · Everyone's network differs
Simulcast & adaptive bitrate
One participant is on gigabit fiber, another on a spotty phone. If the SFU forwards the same high-quality stream to both, the phone user's video freezes and stutters. But the sender uploads only one stream — how can each receiver get a quality that fits their network?
Design decision: The sender uploads one stream, but receivers have wildly different bandwidth. How does each get a fitting quality?
The call: Simulcast: the sender uploads several quality layers; the SFU forwards the layer that fits each receiver's bandwidth. — With simulcast (or SVC) the sender encodes and uploads a few resolutions/bitrates simultaneously; the SFU, watching each receiver's reported bandwidth, forwards the highest layer that fits. One upload effort, per-receiver quality — the phone gets low-res, the fiber gets HD, and the SFU adapts in real time.
Use simulcast (or scalable video coding, SVC): the sender uploads a few quality layers at once (e.g. low/medium/high). The SFU, tracking each receiver's available bandwidth (via congestion feedback), forwards the highest layer that fits each receiver — HD to the fiber user, low-res to the phone. And it can drop to audio-only when needed. One upload, adaptive per-receiver quality, all decided at the forwarding server.
Adapt at the SFU: The SFU is the perfect place to adapt because it sees every receiver's conditions. Simulcast means the sender does the work once (a few layers) and the SFU makes the cheap choice of which layer to forward to whom, reacting to congestion in real time. This is why one person's bad wifi no longer degrades everyone.
Step 6 · Just connecting is hard
Signaling & NAT traversal
Before any media flows, two peers on the internet — both behind home routers (NAT) with no public address — have to find a network path to each other and agree on codecs. Neither can just be "dialed." How do they connect at all?
Use a signaling server to exchange setup info: SDP offer/answer (codecs, parameters) and ICE candidates (possible network paths). To pierce NAT: STUN tells a peer its own public address so peers can try a direct connection; when a direct path is impossible (strict/symmetric NAT, firewalls), TURN relays the media through a server as a fallback. Signaling only sets up the call; once connected, media flows peer↔SFU directly.
Signaling ≠ media: Signaling is the phone book and handshake (who, what codecs, which paths) — it carries no media. ICE tries candidates in order (direct → STUN-assisted → TURN relay) and picks the best that works. TURN is the always-works fallback that also costs the most (it relays all media), so you use it only when nothing else connects.
Step 7 · Big and global calls
Geo-distributed & cascaded SFUs
A single SFU has limits — its bandwidth caps how many streams it can forward, and a call with participants on different continents shouldn't route everyone through one distant server. How do you scale to huge, global calls?
Distribute SFUs geographically so each participant connects to a nearby one (low latency), and cascade them: SFUs peer with each other so a stream crosses between regions once rather than every remote participant pulling it across the ocean. For very large calls (webinars, thousands of viewers), forward only what matters — active-speaker selection and a few top video streams — instead of everyone's stream to everyone. The topology becomes a tree of SFUs, not one hub.
Cascade + active speaker: Cascading SFUs turns one overloaded hub into a mesh of regional servers, each serving local peers and exchanging streams between regions efficiently. At huge scale you also stop forwarding all N videos: send the active speaker(s) and a handful of others — humans can only watch a few faces anyway.
Step 8 · The sharp edges
Loss, congestion, echo & recording
Real networks are hostile: packets drop, bandwidth fluctuates, audio echoes, and you often need to record the call — all while keeping latency low.
Survive packet loss with a jitter buffer (smooth timing), NACK (selective retransmit of just-lost packets when time allows), and FEC (forward error correction — send redundancy so small losses self-heal). Run congestion control (estimate available bandwidth and tell simulcast which layer to send) so the call adapts instead of collapsing. Do echo cancellation / noise suppression on the client audio. And record by attaching a special subscriber that mixes and transcodes the streams to storage — off the live path.
Design for the unhappy path: Loss → jitter buffer + NACK + FEC. Bandwidth swings → congestion control drives simulcast. Echo → client DSP. Recording → a subscriber that mixes/transcodes to storage. Real-time media is a constant negotiation with an unreliable network; these mechanisms are how the call stays watchable when the network isn't.
You did it
You just designed a video conferencing system.
- Real-time media values timeliness over reliability — use UDP/WebRTC, drop late packets, not TCP.
- A direct peer-to-peer connection is ideal for a 1:1 call (lowest latency, no server).
- A full mesh is O(N²) and forces each peer to upload their stream N-1 times — it collapses at ~5 people.
- An SFU lets each peer upload once and forwards streams (no transcoding) — the scalable topology.
- Simulcast + adaptive bitrate let the SFU send each receiver a quality that fits its network.
- Signaling (SDP) + ICE/STUN/TURN connect peers across NATs; TURN relays when direct fails.
- Geo-distributed, cascaded SFUs + active-speaker selection scale to huge, global calls; loss/congestion handled with jitter buffers, NACK, FEC.