Design TikTok: Short Video Feed, Upload Pipeline and Recommendations

Problem Statement

Design a short-video platform like TikTok with an infinite-scroll "For You" feed, video creation/upload with effects, a recommendation algorithm that learns from user behavior in real time, and global video delivery. Must handle 1B+ MAU with engagement-optimized content ranking.

Requirements

Functional

• Upload short videos (15s-3min) with filters, music overlay, and effects
• "For You" feed: personalized infinite scroll of recommended videos from all creators
• Following feed: videos from followed creators, reverse chronological
• Interact: like, comment, share, duet/stitch (respond to another video)

Non-Functional

• Latency: Next video preloaded and starts instantly (0ms perceived wait) during scroll
• Scale: 1.5B MAU, 1B DAU, 1B video views/hour, 5M new videos uploaded/day
• Recommendation quality: Users average 90+ minutes/day -- cold start for new users in <10 videos
• Global: Low-latency delivery to 150+ countries

Core Architecture

1. Video Processing Pipeline -- User uploads video to blob storage. An async pipeline: (1) validates format/length, (2) applies server-side effects if needed, (3) transcodes to multiple bitrates (360p/720p/1080p) in H.264 and H.265, (4) generates thumbnail and preview, (5) runs content moderation (nudity, violence, copyright via audio fingerprinting). Completed videos enter the recommendation candidate pool.

2. For You Recommendation Engine -- Two-phase: (a) Candidate retrieval selects ~5000 videos from collaborative filtering (similar user behavior), content features (hashtags, audio, visual embeddings), and trending pool. (b) Ranking model predicts engagement probability using features: video quality, creator authority, user interest vector, time-of-day, and diversity requirements. Re-ranks in real time as user engagement signals arrive.

3. Video CDN with Prefetch -- Videos are distributed to CDN edge nodes globally. The client prefetches the next 3-5 videos in the feed while the user watches the current one. Popular videos are proactively pushed to edges; long-tail videos use pull-through caching. This ensures zero-wait transitions between videos.

4. Real-time Engagement Pipeline -- Every interaction (watch time, like, skip, share, replay) is sent to Kafka within 100ms. A Flink streaming job updates the user's interest vector and feeds signals back to the recommendation engine. This creates a tight feedback loop: if a user watches 3 cooking videos in a row, the 4th recommendation shifts toward cooking content.

Database Choice

Cassandra for the user interaction log (likes, watch history) -- write-heavy, time-series, partitioned by user_id. PostgreSQL for video metadata, creator profiles, and follow relationships. Redis for the precomputed feed cache (list of next 100 video_ids per user) and real-time engagement counters. S3/GCS for video storage. Feature Store (Redis/DynamoDB) for serving ML features at inference time with sub-5ms latency.

Key API Endpoints

GET /api/v1/feed/foryou?cursor=\{token\}&limit=10
  -> Returns: \{ videos: [\{ video_id, cdn_url, creator, description, music, stats \}], next_cursor \}

POST /api/v1/videos/upload (resumable)
  -> Returns: \{ video_id: "V-789", status: "PROCESSING" \}

POST /api/v1/interactions
  -> Body: \{ video_id: "V-789", type: "watch_time", value_ms: 14200 \}

Scaling Insight

The real-time feedback loop is TikTok's competitive advantage. Traditional recommendation systems update user profiles in batch (hourly/daily). TikTok's Flink pipeline updates the user's interest vector within seconds of each interaction. This means the system adapts to user mood and intent within a single session -- watch 2 videos about dogs, and the 3rd recommendation is already about dogs. This tight loop is why new users get hooked within minutes despite having no history.

Key Tradeoffs

Original DosiBridge notes for Design TikTok: Short Video Feed, Upload Pipeline and Recommendations

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Design TikTok: Short Video Feed, Upload Pipeline and Recommendations is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Core Entity: For Design TikTok: Short Video Feed, Upload Pipeline and Recommendations, decide the core entity before choosing tools; otherwise the design looks clean but fails when tiktok changes.
• Read/Write Path: For Design TikTok: Short Video Feed, Upload Pipeline and Recommendations, decide the read/write path before choosing tools; otherwise the design looks clean but fails when tiktok changes.
• Contention Point: For Design TikTok: Short Video Feed, Upload Pipeline and Recommendations, decide the contention point before choosing tools; otherwise the design looks clean but fails when tiktok changes.
• Abuse Path: For Design TikTok: Short Video Feed, Upload Pipeline and Recommendations, decide the abuse path before choosing tools; otherwise the design looks clean but fails when tiktok changes.
• Recovery Workflow: For Design TikTok: Short Video Feed, Upload Pipeline and Recommendations, decide the recovery workflow before choosing tools; otherwise the design looks clean but fails when tiktok changes.

Design TikTok: Short Video Feed, Upload Pipeline and Recommendations correctness boundary

For this problem, separate the user-visible action from background work. The correctness boundary is the smallest state transition that must not be duplicated, lost, or reordered. Naming that boundary makes the solution more original than a stock API/database/cache template.

Design TikTok: Short Video Feed, Upload Pipeline and Recommendations scaling bottleneck

The first bottleneck should be described in the language of this system: a hot timeline, a contested seat, a busy wallet ledger, a media-processing queue, or a search index update delay. That bottleneck decides which component deserves the deepest interview discussion.

Production review question

If Design TikTok: Short Video Feed, Upload Pipeline and Recommendations suddenly receives 10x traffic, which entity becomes contested first, and what queue, lock, shard, or cache protects that entity without hiding correctness bugs?

Practical metric to track

Track tiktok health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Design TikTok: Short Video Feed, Upload Pipeline and Recommendations, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• Design TikTok -- Reference
• Source: System-Design-Overview