Full-Text Search

Full-text search lets users query large volumes of unstructured text and get results ranked by relevance, not just exact matches. It works by building inverted indexes that map every word to the documents containing it, enabling sub-second lookups across millions of records. Technologies like Elasticsearch, Apache Solr, and PostgreSQL's tsvector power search features in virtually every modern application, from e-commerce product search to log analysis.

Why This Matters

Full-text search is fundamental because nearly every user-facing application requires some form of search. A naive LIKE '%term%' SQL query scans every row and cannot rank results by relevance — it's unusable at scale. Full-text search engines solve this with inverted indexes that turn the problem inside out: instead of scanning documents for words, they look up words to find documents. Understanding how tokenization, stemming, scoring, and index management work is critical for designing systems that return the right results fast while keeping indexes fresh and storage manageable.

The Building Blocks

• Inverted Index: The core data structure that maps each unique term to a list of document IDs containing it, enabling O(1) term lookups instead of full table scans.
• Tokenization & Analysis: Text is split into tokens, lowercased, stripped of stop words, and stemmed (e.g., 'running' → 'run') to normalize matching across different word forms.
• Relevance Scoring: Algorithms like BM25 and TF-IDF rank documents by how frequently and uniquely a term appears — a word in 3 documents scores higher than one in 3 million.
• Analyzers & Mappings: Custom analyzers define how text is processed — language-specific stemming, synonym expansion, n-gram tokenization for autocomplete, or phonetic matching.
• Sharding & Replication: Search indexes are sharded across nodes for parallelism and replicated for fault tolerance, allowing horizontal scaling of both indexing and query throughput.

Under the Hood

When a document is indexed, the analysis pipeline first tokenizes the raw text into individual terms. Each term passes through a chain of filters — lowercasing, stop word removal, stemming or lemmatization — producing normalized tokens. These tokens are stored in an inverted index: a dictionary where each term points to a posting list containing document IDs, term frequencies, and positional data.

At query time, the same analysis pipeline processes the search query, producing normalized tokens that are looked up in the inverted index. The search engine retrieves the posting lists for each query term, intersects or unions them based on boolean logic, and scores each matching document. BM25, the standard scoring algorithm, considers term frequency (how often the term appears in the document), inverse document frequency (how rare the term is across all documents), and field length normalization.

For scale, the index is divided into shards distributed across nodes. Each shard processes the query independently and returns its top results. A coordinating node merges these partial results into a globally ranked list. Techniques like segment merging, doc values for sorting, and filter caching ensure queries stay fast even as the index grows to billions of documents.

How Companies Actually Do This

Elasticsearch at GitHub Powers code search across 200+ million repositories, using custom analyzers for code tokenization (splitting on camelCase, dots, underscores) and sharding indexes by repository to parallelize queries.

Amazon Product search on Amazon uses a full-text search engine with synonym expansion, spell correction, and personalized ranking to handle millions of queries per second across a catalog of hundreds of millions of products.

Slack Uses Elasticsearch to index billions of messages across millions of workspaces, enabling instant search with access control filters that ensure users only see messages from channels they belong to.

Common Pitfalls

1. Not keeping search indexes in sync with the source database — stale indexes cause users to find deleted items or miss new ones; use change data capture or transactional outbox patterns
2. Over-indexing fields that don't need full-text search — indexing numeric IDs or timestamps as analyzed text wastes storage and slows indexing without improving search quality
3. Ignoring relevance tuning — default BM25 scoring rarely matches business needs out of the box; you need field boosting, function scoring, and user feedback loops to return useful results

Interview Questions Worth Practicing

1. How would you design a search system that stays in sync with a relational database containing millions of products?
2. Explain how an inverted index works and why it's faster than SQL LIKE queries for text search.
3. How would you handle multi-language search in a system serving users in 20+ countries?

The Tradeoffs

• Storage vs Speed: Inverted indexes duplicate data from the source, consuming significant disk space, but enable sub-second queries across millions of documents
• Freshness vs Throughput: Real-time indexing keeps results current but reduces write throughput; batch indexing is faster but introduces a delay before new content is searchable
• Precision vs Recall: Strict matching returns fewer but more accurate results; stemming and synonym expansion return more results but may include less relevant ones

How to Explain This in an Interview

Here is how I would explain Full-Text Search in a system design interview:

In an interview, start by explaining the inverted index — it's a map from every unique word to the list of documents containing that word, which flips the search problem from scanning all documents to a fast dictionary lookup. Then explain the analysis pipeline: text is tokenized, lowercased, stemmed, and stripped of stop words before indexing. At query time, the same pipeline runs on the user's query so terms match consistently. Results are ranked using BM25, which scores documents higher when they contain rare terms frequently. For scale, mention sharding the index across nodes so queries run in parallel. Finish by noting the key operational challenge: keeping the search index in sync with the source of truth, typically solved with change data capture or an event-driven pipeline.

Related Topics

• Database Indexes
• Database Types
• Caching Strategies
• NoSQL Data Modeling

Original DosiBridge notes for Full-Text Search

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Full-Text Search is to connect the concept to a retrieval path where users judge quality by relevance, latency, and how quickly updates become searchable. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Query Shape: For Full-Text Search, decide the query shape before choosing tools; otherwise the design looks clean but fails when full changes.
• Write Amplification: For Full-Text Search, decide the write amplification before choosing tools; otherwise the design looks clean but fails when text changes.
• Replication Lag: For Full-Text Search, decide the replication lag before choosing tools; otherwise the design looks clean but fails when search changes.
• Migration Safety: For Full-Text Search, decide the migration safety before choosing tools; otherwise the design looks clean but fails when full changes.
• Data Ownership: For Full-Text Search, decide the data ownership before choosing tools; otherwise the design looks clean but fails when text changes.

Production review question

If a team adopts Full-Text Search, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track full-search health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Full-Text Search, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.