Design YouTube (Video Streaming)

A video platform (YouTube, Netflix, TikTok) lets anyone upload a video and lets everyone else watch it smoothly on any device over any network. On the surface it looks like the news feed turned up — extremely read-heavy, globally distributed. But video changes the shape of the problem: the “records” are gigabyte blobs, not kilobyte rows, so the system is bandwidth-bound, not QPS-bound. The dominant cost isn’t database queries — it’s pushing petabytes out of CDNs every day. We’ll follow the framework.

1. Requirements

Functional

• Upload a video (large file, resumable).
• Transcode it into multiple resolutions/bitrates for playback on any device and network.
• Stream playback that adapts to changing bandwidth without buffering.
• Browse/search, view counts, likes, comments.

Non-functional

• Massively read-heavy — a popular video is uploaded once and watched billions of times.
• Smooth playback — fast startup (low time-to-first-frame) and no stalls mid-stream.
• Durability — never lose an uploaded original.
• Global low latency — a viewer in São Paulo and one in Seoul both start instantly.

The read:write ratio is extreme — but the reads are huge

Uploads are rare; views are constant — easily a millions-to-one ratio. That screams “cache everything.” But unlike a feed, each read ships megabytes per second of video, so the bottleneck isn’t request rate, it’s egress bandwidth. This one fact reroutes the entire design toward CDNs and pre-computed renditions instead of databases and replicas.

2. Back-of-envelope estimation

Two numbers matter: how much we ingest, and how much we serve.

INGEST (illustrative, hedged — figures drift)
  uploads:        ~500 hours of video uploaded per minute (oft-cited, ~recent years)
                  = 500 × 60 × 24 ≈ 720,000 hours/day ingested
  each source hour fans out into a "ladder" of ~6 renditions → multiplies stored bytes

EGRESS (this is the whole system)
  watch time:     ~1 billion hours watched/day (oft-cited)
  avg concurrent: 1e9 hours ÷ 24 h ≈ 42M streams being watched at any instant
  per stream:     ~2.5 Mbps average (mixed mobile/desktop)
  egress:         42e6 × 2.5e6 bits/s ≈ 1.0e14 bits/s ≈ ~100 Tbps average
                  peak 2–3× → hundreds of Tbps

A hundred-plus terabits per second, sustained, is a quantity no origin datacenter serves directly. It is served from thousands of CDN edge caches close to viewers. The origin’s job shrinks to producing renditions and seeding the CDN; the CDN does the heavy lifting. Compare the modest write path (720k upload-hours/day) against the colossal read path — the asymmetry is the design.

3. API sketch

Uploads use a resumable protocol because a multi-gigabyte POST that dies at 90% must not restart from zero.

POST /v1/videos                       → { video_id, upload_url }   // reserve, get resumable URL
PUT  {upload_url}  (Content-Range: bytes 0-...)                    // chunked, resumable
POST /v1/videos/{id}/complete         → { status: "processing" }   // triggers transcode pipeline

GET  /v1/videos/{id}                  → { title, manifest_url, ... } // metadata + playback manifest
GET  /v1/videos/{id}/master.m3u8      → ABR manifest (variant list)  // served from CDN
POST /v1/videos/{id}/view             → 204                         // fire-and-forget view event

The client never asks for “the video file.” It fetches a manifest that lists the available bitrate renditions, then pulls small segments on demand — the heart of adaptive streaming.

4. Data model

The defining split: metadata in a database, bytes in object storage.

videos (SQL/NoSQL row — small)        segments (object storage — huge blobs)
  video_id     (PK)                     s3://bucket/{video_id}/1080p/seg_0001.ts
  uploader_id                           s3://bucket/{video_id}/720p/seg_0001.ts
  title, description                    ...  (immutable, CDN-cacheable)
  status (processing|ready)
  manifest_url                        view_counts (write-heavy counter — see §6)
  duration, created_at                  video_id (PK)
                                        count   (approximate, batched)

Never put video bytes in a relational database. The metadata (titles, owners, status) is small, queried richly, and belongs in a database; the media is enormous, immutable once transcoded, and belongs in object storage fronted by a CDN. Segments are content-addressable and cacheable forever — exactly what an edge cache wants.

5. High-level design — the transcoding pipeline

 UPLOAD ─► resumable upload service ─► raw original ─► OBJECT STORE (source of truth, durable)
                                                          │
                                                          ▼  enqueue transcode job
                                                  ┌────────────────┐
                                                  │  Message queue  │  (split into chunks)
                                                  └───────┬────────┘
                                  ┌───────────────┬───────┴───────┬───────────────┐
                                  ▼               ▼               ▼               ▼
                            transcode         transcode       transcode       transcode   (worker fleet)
                            chunk→144p..4K     ...             ...             ...
                                  │  package into HLS/DASH segments + manifests
                                  ▼
                            packaged renditions ─► seed CDN edges ─► READY
                                                          │
   PLAYBACK:  player ─► GET master manifest (CDN) ─► picks bitrate ─► GET segments (CDN) ─► decode

Transcoding is embarrassingly parallel: split the source into independent chunks, fan them out to a worker fleet through a message queue, transcode each into the full bitrate ladder in parallel, then stitch the segments and write manifests. The poster gets an instant processing response; the video appears once the pipeline finishes. This is the same async-fan-out pattern as the web crawler — a queue decouples the slow, spiky work from the fast request.

Under the hood — ABR, HLS, and DASH

Adaptive bitrate (ABR) is how playback survives a flaky network. The transcoder produces the same video at several resolutions/bitrates — the bitrate ladder (e.g. 144p, 240p, 360p, 480p, 720p, 1080p, 4K), and chops each into short segments (typically 2–10 seconds). A manifest lists them:

master.m3u8 (HLS) or .mpd (DASH)        ← lists the bitrate variants
  ├── 240p/index.m3u8  → seg_0001.ts, seg_0002.ts, ...
  ├── 720p/index.m3u8  → seg_0001.ts, seg_0002.ts, ...
  └── 1080p/index.m3u8 → seg_0001.ts, seg_0002.ts, ...

The client measures its download speed and buffer level and picks the next segment’s bitrate on the fly — bandwidth drops, it grabs the 480p segment instead of 1080p; bandwidth recovers, it climbs back up. HLS (HTTP Live Streaming, from Apple, .m3u8) and MPEG-DASH (.mpd, an ISO standard) are the two dominant ABR protocols; both ride plain HTTP, which is precisely why ordinary CDNs can cache the segments. Segments are just files — no special streaming server required.

By the numbers — why the CDN is the whole architecture

The egress estimate said ~100 Tbps average. Suppose, hypothetically, you tried to serve that from origin. Even at a generous 100 Gbps per server NIC:

100 Tbps ÷ 100 Gbps/server = 1,000 servers just to push bits — if every byte were uncached
plus the cross-continent latency: a viewer in Brazil pulling from a US origin adds ~150 ms RTT/segment

Now front it with a CDN. A popular video’s segments are cached at the edge after the first request in each region, so 99%+ of bytes never touch origin — they ship from a city near the viewer at single-digit millisecond latency. The CDN buys you slashed origin bandwidth and low startup latency; it costs you egress fees (the dominant line item) and a cache that’s only warm for popular content. The long tail — videos watched a few times — barely caches, so its bytes do hit origin. That long-tail egress is why video economics live and die by CDN efficiency.

6. Counting views without melting a counter

A viral video gets millions of view events per minute, all incrementing one row — a textbook hot partition. A synchronous UPDATE ... SET count = count + 1 on a single key would serialize behind a lock and collapse. Instead:

• Fire-and-forget view events into a queue; the player doesn’t wait.
• Batch + aggregate — workers count events in windows and apply periodic bulk increments, so one row gets a few writes per second, not millions.
• Approximate is fine — the displayed count can lag and be probabilistic; nobody needs the 4,001,237th view to be exact in real time. This is eventual consistency, traded deliberately for write throughput.

It actually happened — “Gangnam Style” overflowed YouTube’s view counter (2014)

YouTube originally stored view counts as a signed 32-bit integer, whose maximum value is 2,147,483,647 (~2.15 billion). In late 2014, Psy’s Gangnam Style became the first video to cross that line — and YouTube announced it had upgraded the counter to a 64-bit integer (max ~9.2 quintillion) to keep counting. It’s a perfect, real-world reminder that “just a counter” is a real design decision: pick the wrong width and the most-watched video on Earth literally wraps your column. The fix — wider integers, batched/approximate increments — is the same medicine this section prescribes.

Key trade-offs

• CDN vs origin: the CDN buys low-latency global delivery and slashed origin bandwidth; it costs egress fees and only helps popular (cacheable) content — the long tail still hits origin.
• Pre-transcode every rendition vs transcode on demand: pre-computing the full ladder buys instant, cache-friendly playback at the cost of storage and compute spent on renditions nobody may watch; on-demand saves storage but adds first-view latency.
• Bytes in object store vs database: separating media (blobs) from metadata (rows) buys the right tool for each — cheap durable storage vs rich queries — at the cost of a two-system data model.
• Exact vs approximate view counts: approximate, batched counting buys write throughput on hot videos; it costs real-time precision nobody actually needs.

Check your understanding

1. Why is a video platform described as bandwidth-bound rather than QPS-bound, and how does that change the design compared to a news feed?
2. Walk through the transcoding pipeline. Why is it modeled as parallel chunks behind a queue?
3. What is adaptive bitrate streaming, and what role do the manifest and segments play? Why can an ordinary HTTP CDN cache them?
4. Why store video bytes in object storage but metadata in a database?
5. A single viral video receives millions of view events per minute. Why would a synchronous counter fail, and what three techniques fix it?

Show answers

1. Each read ships megabytes per second of video, so the limiting resource is egress bandwidth, not request rate. A news feed serves tiny rows and is designed around databases, caches, and replicas; a video platform serves huge immutable blobs and is designed around CDNs and pre-computed renditions — ~100 Tbps of average egress is served from thousands of edge caches, not from origin databases.
2. Upload lands the raw original in durable object storage, which enqueues a transcode job; workers split the source into independent chunks, transcode each into the full bitrate ladder in parallel, then package HLS/DASH segments + manifests and seed the CDN. It’s modeled as parallel chunks behind a queue because transcoding is embarrassingly parallel and slow — the queue decouples the spiky, expensive work from the instant processing response the uploader gets.
3. ABR produces the same video at several bitrates (the ladder), each chopped into short segments; a manifest lists the variants and their segments. The client measures bandwidth/buffer and picks each next segment’s bitrate on the fly, climbing down on congestion and back up on recovery. Because HLS (.m3u8) and DASH (.mpd) ride plain HTTP and segments are just files, ordinary CDNs cache them — no special streaming server needed.
4. Media is enormous, immutable once transcoded, and best served cheaply and durably from object storage fronted by a CDN; metadata (titles, owners, status) is small and richly queried, which is what a database is for. Putting gigabyte blobs in a relational DB would be slow, costly, and pointless. The cost of the split is maintaining a two-system data model.
5. All those events increment one row — a hot partition that serializes behind a lock and collapses under synchronous +1 writes. Fix: fire-and-forget events into a queue (player doesn’t wait), batch/aggregate increments in windows so the row sees a few writes/sec, and accept an approximate, eventually-consistent count — trading real-time precision (which nobody needs) for write throughput.