The whole design, in writing
Learn system design by building a video platform like YouTube step by step. An interactive guide covering metadata vs blob storage, the async transcoding pipeline and ABR ladder, CDN edge delivery, tiered hot/cold storage, scaling the control plane, async watch analytics, and resumable uploads, Content-ID and live.
Every step of the build above, written out: the problem each piece solves, the option that was taken and the ones that were not, the numbers, and how it fails in production.
The big idea
How does YouTube stream to a billion screens?
A creator uploads one enormous file. Billions of viewers — on fibre, on 3G, on a TV — must each watch it starting in under a second, in the best quality their connection can handle, without buffering.
The trick is that you never serve "the video." You serve a metadata lookup, then a flood of tiny pre-encoded segments from a server near the viewer. Almost everything here is about turning one raw upload into something a CDN can spray across the planet.
What the new pieces do
- Upload UIcreator
- A creator uploading a huge raw file. Uploads once and wants it watchable everywhere, in every quality.
- Playerviewer
- Anyone hitting play. Expects the video to start in under a second and never buffer, on any connection.
Step 1 · The skeleton
Split the bytes from the facts
A video is two very different things: a few hundred bytes of metadata (title, owner, status) and gigabytes of raw media. Storing them the same way is a mistake.
A video is a few hundred bytes of metadata plus gigabytes of raw media. How do you store them?
Gigabyte media blobs bloat a database, make queries slow, and waste its expensive indexed storage. Databases are for small structured queryable data, not opaque write-once files.
Object storage is great for the giant media but terrible for the queryable facts (title, owner, status, search) you constantly look up. You need a database for those.
Tiny structured facts go in a queryable database; huge write-once media goes in cheap object storage. Watch requests read metadata first, then fetch bytes — splitting them is the foundation.
Keep facts in a Metadata DB and bytes in a Blob Store (object storage like S3). The Upload Service streams the raw file into the blob store and writes a small row in the metadata DB. Watch requests read metadata first, then go get bytes.
What the new pieces do
- App Serveredge
- The front door for uploads and watch requests. Becomes an API Gateway behind a load balancer.
- Upload Serviceservice
- Accepts the raw upload (resumable, in chunks), stashes it in blob storage, and queues it for processing.
- Metadata DBstore
- Small structured data: title, description, owner, status, and the manifest of available renditions.
- Blob Storestore
- Object storage (S3-like) holding the giant raw upload and every encoded segment. The origin behind the CDN.
Step 2 · The pipeline
One file becomes many
The raw upload is useless for streaming: it’s one giant file in one codec at one resolution. A phone on 3G and a 4K TV need wildly different versions.
The raw upload is one giant file in one codec. A 3G phone and a 4K TV need different versions. How do you produce them?
Re-encoding per view is enormously expensive and slow — you’d redo the same work for every viewer. Encode once, up front, into all qualities and reuse.
The upload service drops a job on a queue; a transcoder fleet re-encodes into 240p–4K, each chopped into small ~2–10s segments written back to blob, then marks the video ready. The player switches quality mid-stream from the precomputed ladder (HLS/DASH).
Creators upload one master file and can’t be expected to produce a full ABR ladder of segmented renditions. Encoding is the platform’s job, done asynchronously off the upload path.
Make encoding asynchronous. The Upload Service drops a job on a queue; a fleet of Transcoder workers re-encode the file into an ABR ladder — 240p, 480p, 720p, 1080p, 4K — each chopped into small (~2–10s) segments, then written back to the blob store. When done, they mark the video ready in metadata.
- 5–6quality rungs
- ~4sper segment
- asyncvia queue
What the new pieces do
- Transcoderworker
- Workers that re-encode the raw file into many resolutions and bitrates, split into small streamable segments.
- Encode Jobsbus
- A job queue decoupling fast uploads from slow encoding. Transcoder workers pull jobs at their own pace.
Back of the envelope
- 1 upload → 5–6 rungs × segments
- ~6× storage per video
- ~2–10s per segment
- switchable mid-stream (HLS/DASH)
- encode async via queue
- slow encoding off the fast upload path
Step 3 · The hot path
Serving the watch
Views dwarf uploads by orders of magnitude, and viewers are everywhere on Earth. Streaming every segment from one origin datacenter would be slow and ruinously expensive.
Views dwarf uploads and viewers are everywhere. Streaming every segment from one origin would be slow and ruinous. Fix?
More origin capacity in one place still pays a long round-trip to distant viewers and concentrates bandwidth cost. You need bytes physically close to users.
Pre-downloading gigabytes wastes bandwidth (most of a video is never watched) and delays start. You want to stream small segments on demand from a nearby edge.
The player pulls the manifest then segments from the nearest edge; the first regional viewer warms the cache, everyone after gets bytes from down the street. Immutable segments cache perfectly — the CDN offloads 95%+ of traffic. The CDN IS the read-scaling strategy.
Put a CDN in front of the blob store. To watch, the player fetches the manifest (from metadata) then pulls segments from the nearest CDN edge. The first viewer in a region warms the cache; everyone after gets bytes from down the street. Origin traffic stays tiny.
- 95%+served at edge
- <1sstart time
- immutablesegments
What the new pieces do
- CDN Edgeedge
- Thousands of edge servers near users. Serves the actual video bytes so the origin barely gets touched.
Back of the envelope
- 95%+ served at the edge
- origin barely touched
- immutable segments
- cache forever, no invalidation
- 1st regional view warms cache
- the rest served locally, <1s start
Step 4 · A mountain of bytes
Storing exabytes affordably
Every upload now becomes 5–6 renditions, each in many segments — multiplying storage. Hundreds of hours are uploaded every minute. You cannot keep it all hot.
Every upload becomes 5–6 renditions in many segments, and 500 hours arrive every minute. How do you store exabytes affordably?
A tiny fraction of videos drive most views — keep those replicated and ready; let the millions of near-zero-view uploads sit in cheap cold tiers, accepting a slower first byte if someone finally watches them.
Replicating everything everywhere is ruinously expensive when most videos are barely watched. Match storage cost to popularity — hot content hot, cold content cheap.
Deleting creators’ uploads to save space is unacceptable — they expect permanence. The answer is cheaper cold storage for the long tail, not deletion.
Lean on tiered blob storage: replicate popular content across regions for fast origin pulls, and push the long tail of rarely-watched videos to cheaper cold storage. Generate thumbnails and previews once, store them as blobs too, and serve them from the CDN.
- 500 hrsuploaded / min
- 6×storage per video
- tieredhot + cold
Back of the envelope
- 500 hrs uploaded / minute
- × ~6 renditions = exabytes
- hot: replicated + ready
- the tiny fraction driving most views
- cold: cheap, slower first byte
- the long tail nobody watches
Step 5 · Don’t fall over
Scale the control plane
The CDN handles the bytes, but metadata reads, search, and upload coordination still hit your services — and one big server is a single point of failure.
The CDN handles bytes, but metadata reads, search and upload coordination still hit your services, and one server is a SPOF. How do you scale them?
A bigger box has a ceiling and is still a single point of failure, and it can’t isolate an upload spike from the read-heavy watch path. You need many instances and separation of concerns.
Coupling uploads/encoding (write) with watch-metadata/search (read) means a surge in one takes down the other. Their traffic shapes are totally different — isolate them.
Run many stateless instances behind a load balancer with a hot-metadata cache, and separate upload, watch-metadata and search so each scales — and fails — independently, keeping the 99% read path resilient when writes surge.
Put a load balancer in front and run many stateless copies of each service. Add a read cache for hot metadata. Split upload, watch-metadata, and search into services that scale — and fail — independently, so a spike in uploads never takes down playback.
Step 6 · Count the views
Views, watch time & recs
Owners and the recommender want view counts, watch time, and quality stats. Writing a row on every play — and incrementing a counter on a viral video — would hammer the database.
Owners want view counts and watch time, but a DB write per play (and a counter bump on a viral video) would hammer it. How?
Plays emit events into an analytics pipeline; counts aggregate in the background (approximated for hot videos), and recommendations train on the same stream — never on the playback critical path. If analytics lags, playback is unaffected.
A synchronous counter bump per play on a viral video is a write hotspot that can melt the DB and adds latency to playback. Counting must be async and off the hot path.
Client-only counts are unverifiable and easily gamed, and still need to reach the server. Emit events to a pipeline that aggregates and validates server-side.
Make it asynchronous. The player emits lightweight watch events that flow into an analytics pipeline; counts are aggregated in the background and approximated for hot videos. Recommendations train on this same event stream — never on the playback critical path.
What the new pieces do
- View Analyticsstore
- Aggregates views, watch time and quality switches asynchronously, so counting a view never slows playback.
Step 7 · The sharp edges
Uploads, copyright & live
Real life intrudes: uploads fail halfway on flaky connections, pirated content appears, and some events must stream live with seconds of delay, not after a full encode.
Use resumable, chunked uploads so a dropped connection retries only the missing chunk. Run a content-matching system (fingerprinting) to flag copyrighted material. For live, encode segments on the fly and publish the manifest continuously — same ABR + CDN idea, just streaming as it happens.
You did it
You just designed YouTube.
Everything you assembled, in order
- Split tiny metadata (DB) from huge media (blob store).
- Async transcoding turns one raw file into an ABR ladder of segments.
- Adaptive bitrate lets the player switch quality without buffering.
- A CDN serves immutable segments from the edge — 95%+ offload.
- Tiered hot/cold blob storage keeps exabytes affordable.
- Async watch events drive view counts and recommendations.
- Resumable uploads, content-ID, and live streaming for the real world.