Design YouTube: Video Upload, Transcoding, CDN and Recommendations

Problem Statement

Design a video sharing platform like YouTube supporting video upload, transcoding to multiple resolutions and formats, adaptive bitrate streaming, accurate view counting at massive scale, comments, likes, and a recommendation system. Must handle 500 hours of video uploaded per minute and 1B hours watched daily.

Requirements

Functional

• Upload videos (up to 12 hours); transcode to multiple resolutions (144p to 4K) and formats (H.264, VP9, AV1)
• Adaptive bitrate streaming: client switches resolution seamlessly based on bandwidth
• Accurate view counting with deduplication (no bots, no double-counts)
• Personalized video recommendations on the home page and "Up Next" sidebar

Non-Functional

• Latency: Video playback starts within 2 seconds; uploads processed within 30 minutes for most videos
• Scale: 2B MAU, 500 hours of video uploaded/minute, 1B hours watched/day
• Storage: ~1 exabyte of video content
• Availability: 99.99% for video playback

Core Architecture

1. Upload and Transcoding Pipeline -- User uploads original video to a staging bucket. A Kafka event triggers a DAG of transcoding jobs: split video into segments, transcode each segment in parallel across multiple resolutions/codecs (H.264 for compatibility, VP9/AV1 for efficiency), generate thumbnails, extract audio tracks, create DASH/HLS manifests. Completed segments are written to the video CDN origin (S3/GCS).

2. Video CDN and Streaming -- Videos are served as DASH/HLS adaptive streams. The CDN caches popular videos at edge nodes. The video player client monitors buffer health and switches between resolution renditions mid-stream. For less popular videos, CDN pull-through fetches segments from origin on demand.

3. View Count Service -- Views are recorded in Kafka with deduplication: same user + same video within 30 seconds = 1 view. A Flink streaming job aggregates view counts in real time. Approximate counts (HyperLogLog for unique viewers) are shown immediately; exact counts are reconciled hourly via batch. This is critical for monetization accuracy.

4. Recommendation Engine -- Two-stage system: (1) Candidate Generation retrieves ~1000 videos from the user's watch history, subscriptions, and collaborative filtering embeddings. (2) Ranking model scores candidates using features like watch time prediction, click-through rate, and diversity. Served via a low-latency inference service (<50ms per request).

Database Choice

Vitess (sharded MySQL) for video metadata, channels, and user data -- Google uses Vitess internally for YouTube's relational data. Bigtable/Cassandra for view counts and watch history -- write-heavy, wide-column, partitioned by video_id or user_id. S3/GCS for video segment storage (exabyte scale). Redis for real-time view count approximations and session data. Elasticsearch for video search (title, description, captions).

Key API Endpoints

POST /api/v1/videos/upload (resumable upload)
  -> Headers: Content-Range, Upload-ID
  -> Returns: \{ video_id: "V-123", status: "PROCESSING", estimated_time_min: 15 \}

GET /api/v1/videos/\{video_id\}/manifest.mpd
  -> Returns: DASH manifest with available resolution renditions

GET /api/v1/recommendations?context=home&limit=20
  -> Returns: \{ videos: [\{ video_id, title, thumbnail_url, channel, view_count, duration \}] \}

Scaling Insight

Segment-level parallel transcoding is what makes YouTube's upload pipeline feasible. A 1-hour video is split into 10-second segments (360 segments). Each segment is transcoded independently across 6 resolutions and 3 codecs = 6,480 parallel tasks. This is embarrassingly parallel and can be distributed across thousands of worker nodes. A 1-hour video that would take 6+ hours to transcode sequentially finishes in ~10 minutes with 600 parallel workers.

Key Tradeoffs

Deep-Dive: Clarifying Questions for YouTube

1. What is the upload volume? 500 hours of video are uploaded to YouTube every minute. That is 720,000 hours per day.
2. What is the watch volume? Users watch over 1 billion hours of video per day across 2 billion logged-in users per month.
3. How does the transcoding pipeline work? Each uploaded video must be transcoded into multiple resolutions (144p to 8K), multiple codecs (H.264, VP9, AV1), and multiple container formats. A single upload can generate 100+ output files.
4. How does the recommendation engine work? YouTube's recommendation engine drives 70% of all watch time. It uses deep learning models trained on billions of watch sessions.
5. Do we need live streaming? Live streams have fundamentally different requirements: low latency (under 5 seconds glass-to-glass), real-time transcoding, and live chat.
6. Content moderation? YouTube reviews millions of videos per day for policy violations using a combination of ML models and human reviewers.

Specific Functional Requirements

1. Video Upload: Upload videos up to 12 hours long and 256 GB in size, with automatic transcoding to multiple formats
2. Video Playback: Stream video with adaptive bitrate, supporting quality switching from 144p to 8K
3. Search: Search across billions of videos by title, description, tags, and spoken content (auto-generated captions)
4. Recommendations: Personalized video suggestions on the homepage and "Up Next" sidebar
5. Comments and Engagement: Like, dislike, comment, share, save to playlist
6. Channels and Subscriptions: Subscribe to channels, notification bell for new uploads
7. Live Streaming: Real-time video broadcasting with live chat

Specific API Endpoints

POST /api/v1/videos/upload
  Body: multipart (video file, title, description, tags, thumbnail)
  Response: { "video_id": "abc123", "status": "processing", "processing_eta_minutes": 30 }

GET /api/v1/videos/:video_id/watch
  Response: { "manifest_url": "https://cdn.youtube.com/abc123/manifest.mpd", "metadata": { "title": "...", "views": 1234567, "likes": 45678 }, "recommendations": [...] }

GET /api/v1/feed/home?page_token=xyz
  Response: { "videos": [...], "next_page_token": "..." }

GET /api/v1/search?q=system+design&order=relevance&page_token=xyz
  Response: { "results": [...], "total_results": 500000, "next_page_token": "..." }

POST /api/v1/videos/:video_id/comment
  Body: { "text": "Great video!" }
  Response: { "comment_id": "c123", "text": "...", "created_at": "..." }

Specific Data Model

Videos (Bigtable/Spanner)

Video Chunks (Google Cloud Storage/GFS): Each video split into 2-5 second segments, each segment encoded in multiple qualities. Total storage: estimated at 1 exabyte+.

View Counts (Memcache + async flush): Real-time view counting uses in-memory counters that flush to the database periodically. "301+ views" used to appear because YouTube had an anti-fraud threshold before committing view counts.

Recommendation Features (ML pipeline): User watch history, watch duration ratios (watched 90% = strong signal), click-through rates, co-watch patterns. All fed into deep learning models that run in batch (hourly) and real-time (per-request) pipelines.

Specific Back-of-the-Envelope Numbers

Upload pipeline:

• 500 hours of video uploaded per minute = 30,000 hours/hour
• Average video: 10 minutes at 1080p = ~1.5 GB raw
• Each video transcoded into ~100 variants (resolutions * codecs)
• Transcoding compute: 500 hours/min * 100 variants * 5x real-time processing = 250,000 hours of compute per minute

Storage:

• New content: 500 hours/min * 60 * 24 = 720,000 hours/day * ~10 GB/hour (across all variants) = 7.2 PB/day of new encoded video
• Total library: estimated at 800M+ videos, 1+ exabyte of storage

Streaming traffic:

• 1 billion hours watched/day, average bitrate 5 Mbps = 5 billion Gbps-hours/day
• Peak concurrent viewers: estimated 100M+ streams simultaneously

CDN requirements:

• Google's global CDN (Google Edge) serves YouTube content from 90+ countries
• Cache hit ratio for popular videos: 95%+
• Long-tail videos (under 100 views) served from origin storage

Original DosiBridge notes for Design YouTube: Video Upload, Transcoding, CDN and Recommendations

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Design YouTube: Video Upload, Transcoding, CDN and Recommendations is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Core Entity: For Design YouTube: Video Upload, Transcoding, CDN and Recommendations, decide the core entity before choosing tools; otherwise the design looks clean but fails when youtube changes.
• Read/Write Path: For Design YouTube: Video Upload, Transcoding, CDN and Recommendations, decide the read/write path before choosing tools; otherwise the design looks clean but fails when youtube changes.
• Contention Point: For Design YouTube: Video Upload, Transcoding, CDN and Recommendations, decide the contention point before choosing tools; otherwise the design looks clean but fails when youtube changes.
• Abuse Path: For Design YouTube: Video Upload, Transcoding, CDN and Recommendations, decide the abuse path before choosing tools; otherwise the design looks clean but fails when youtube changes.
• Recovery Workflow: For Design YouTube: Video Upload, Transcoding, CDN and Recommendations, decide the recovery workflow before choosing tools; otherwise the design looks clean but fails when youtube changes.

Design YouTube: Video Upload, Transcoding, CDN and Recommendations correctness boundary

For this problem, separate the user-visible action from background work. The correctness boundary is the smallest state transition that must not be duplicated, lost, or reordered. Naming that boundary makes the solution more original than a stock API/database/cache template.

Design YouTube: Video Upload, Transcoding, CDN and Recommendations scaling bottleneck

The first bottleneck should be described in the language of this system: a hot timeline, a contested seat, a busy wallet ledger, a media-processing queue, or a search index update delay. That bottleneck decides which component deserves the deepest interview discussion.

Production review question

If Design YouTube: Video Upload, Transcoding, CDN and Recommendations suddenly receives 10x traffic, which entity becomes contested first, and what queue, lock, shard, or cache protects that entity without hiding correctness bugs?

Practical metric to track

Track youtube health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Design YouTube: Video Upload, Transcoding, CDN and Recommendations, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• Design YouTube -- Reference
• Source: System-Design-Overview