Building In-Video Search at Netflix

Company Context

Netflix manages a vast content library with thousands of titles, each containing hours of video. Internal teams — editors, marketers, quality analysts — need to find specific moments within videos: a particular actor's scene, a specific location, or a visual element. Before in-video search, finding a 3-second clip in a 2-hour film required manually scrubbing through content, a process that could take hours per query.

The Problem at Scale

Traditional search indexes text metadata (title, description, tags), but most of the information in a video exists only in the visual and audio streams. Netflix needed a way to index the content of videos themselves — every frame, every spoken word, every on-screen text — and make it searchable with sub-second query response times across their entire catalog.

Architecture Solution

Netflix built a multi-modal content understanding pipeline that processes videos through several ML models in parallel. A scene detection model segments each video into semantically coherent scenes. Object detection and face recognition models identify people, objects, and locations in each frame. OCR extracts on-screen text (credits, signs, subtitles). ASR (Automatic Speech Recognition) transcribes spoken dialogue with timestamps.

The outputs of these models are combined into a temporal index — a mapping from semantic concepts to precise time ranges within each video. This index is stored in an Elasticsearch cluster optimized for temporal range queries. When a user searches for "beach sunset with two people," the system queries across visual embeddings, recognized objects, and scene descriptions to return time-stamped results.

The processing pipeline runs on a distributed compute platform that schedules ML inference jobs across GPU clusters. New titles are processed as they are ingested, and the catalog is periodically reprocessed as models improve. Each video goes through the pipeline once, and the extracted features are cached for repeated querying.

Key Techniques Used

• Multi-modal feature extraction: Parallel ML models for vision, speech, text, and faces
• Temporal indexing: Time-range annotations linking concepts to specific video segments
• Vector embeddings: Semantic similarity search for visual concepts
• Scene segmentation: Breaking continuous video into searchable semantic units
• Distributed GPU scheduling: Parallel inference across large compute clusters
• Incremental processing: New content indexed on ingestion; existing content reprocessed as models improve

Lessons for System Design Interviews

This case study demonstrates how to design a search system for non-text content. Key principles: decompose complex media into searchable features using specialized ML models, store features in an inverted index with temporal metadata, and separate the offline processing pipeline from the online query path. When asked "design a video search system," reference Netflix's approach of parallel multi-modal extraction feeding into a unified temporal index.

Lessons for Production

ML model accuracy improves over time, so the pipeline must support reprocessing existing content. The biggest cost is GPU compute for inference, so batching and scheduling matter. Pre-computing features offline and serving from an index is far cheaper than running models at query time. Design the index schema to support the queries your users will actually run.

Original DosiBridge notes for Building In-Video Search at Netflix

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Building In-Video Search at Netflix is to connect the concept to a media-heavy path where network variance, buffering, and edge placement decide the user experience. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Practice Loop: For Building In-Video Search at Netflix, decide the practice loop before choosing tools; otherwise the design looks clean but fails when netflix changes.
• Implementation Exercise: For Building In-Video Search at Netflix, decide the implementation exercise before choosing tools; otherwise the design looks clean but fails when in changes.
• Review Checkpoint: For Building In-Video Search at Netflix, decide the review checkpoint before choosing tools; otherwise the design looks clean but fails when video changes.
• Failure Drill: For Building In-Video Search at Netflix, decide the failure drill before choosing tools; otherwise the design looks clean but fails when search changes.
• Comparison Target: For Building In-Video Search at Netflix, decide the comparison target before choosing tools; otherwise the design looks clean but fails when netflix changes.

Production review question

If a team adopts Building In-Video Search at Netflix, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track netflix-search health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Building In-Video Search at Netflix, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• Building In-Video Search — Netflix Technology Blog