Fault Tolerance

The Problem Fault Tolerance Solves

In large-scale systems, component failures are not exceptions — they are the norm. Google reports that in a cluster of 10,000 servers, you can expect about 1,000 individual disk failures per year. Fault tolerance is what keeps these systems running.

How It Works Under the Hood

Fault tolerance is the ability of a system to continue functioning correctly when one or more of its components fail. Unlike failover (which switches to a backup), a fault-tolerant system is designed so that failures are transparent to users — the system handles them internally without any visible disruption.

A fault-tolerant system anticipates failures at every layer. At the hardware level: RAID storage, ECC memory, redundant power. At the network level: multiple paths, BGP failover. At the application level: health checks, circuit breakers, retry logic. At the data level: replication, write-ahead logs, backups.

The design principle: assume anything can fail at any time, and build the system to handle it automatically.

The Mental Model

• Replication: Store multiple copies of data across independent failure domains. If one copy is lost, others remain available.
• Redundancy: Run multiple instances of every service. N+1 redundancy means you have one more instance than needed.
• Isolation: Failures in one component should not cascade to others. Use bulkheads, circuit breakers, and timeouts.
• Graceful degradation: When a component fails, the system reduces functionality instead of crashing entirely. Show cached data instead of an error page.
• Retry with backoff: Transient failures (network blips) can be handled by retrying with exponential backoff and jitter.

Real Systems That Depend on This

Google's GFS: Every file is stored in 3 replicas across different racks. If a server or disk fails, the system automatically re-replicates to maintain the 3-copy guarantee.

Netflix Chaos Engineering: Netflix intentionally injects failures in production (Chaos Monkey, Chaos Kong) to continuously validate fault tolerance.

Cockroach DB: Survives entire data center failures by replicating data across regions with Raft consensus.

Where This Shows Up in Interviews

1. How do you make a system fault-tolerant?
2. What is the difference between fault tolerance and high availability?
3. How do circuit breakers improve fault tolerance?
4. What is graceful degradation?

Tradeoffs

• Cost vs. Resilience: Fault tolerance requires redundant resources — more servers, more storage, more network capacity.
• Latency vs. Durability: Synchronous replication is more fault-tolerant but slower.
• Complexity vs. Reliability: More fault-tolerance mechanisms mean more code to maintain and test.

Watch Out For

1. Not isolating failure domains — replicas on the same rack fail together
2. Infinite retries without backoff — can overwhelm a recovering service
3. Testing only happy paths — never simulating actual failures

Go Deeper

• Availability — start here if this is new to you
• reliability
• failover
• Circuit Breaker
• disaster-recovery

Production angle: fault tolerance is controlled failure, not failure prevention

Fault tolerance does not mean a system never breaks. It means the system keeps the most important user journey alive when one component, zone, dependency, or data path fails. The design question is not "how do we avoid all failure?" but "which failure do we absorb, what do users see, and how do operators recover?"

Fault domains to separate

• Process failure: one instance crashes and traffic moves to healthy replicas.
• Node failure: a host or container runtime dies; orchestration reschedules workloads.
• Zone failure: a datacenter zone loses networking or power; traffic shifts across zones.
• Dependency failure: payment, email, maps, search, cache, or queue becomes unavailable.
• Data failure: corrupted writes, replication lag, schema errors, or accidental deletes.

A mature architecture treats these separately. Running three replicas on the same host is not fault tolerant. Replicating bad data instantly to every region is also not fault tolerant.

Example: checkout during email outage

If the email provider fails during checkout, the order should still be placed. The system can store an 'EmailNotificationPending' job, retry later, and show the user the confirmation page. Email is important, but it should not be a synchronous blocker for taking payment and creating the order. That is fault-tolerant design: preserve the core action and degrade the non-core action.

Design mechanisms

• Redundancy for compute and storage.
• Health checks that remove bad instances quickly.
• Timeouts so callers do not wait forever.
• Circuit breakers to stop cascading failure.
• Queues to absorb temporary downstream outages.
• Idempotency to make retry safe.
• Backups and restore drills for data failure.
• Runbooks and alerts tied to user impact, not just CPU.

Interview-ready tradeoff

Every fault-tolerance mechanism has cost. Multi-region active-active improves availability but adds consistency and operational complexity. Queues improve resilience but add delayed processing. Retries improve transient recovery but can amplify traffic during an outage. The best answer names the critical user journey first, then chooses fault-tolerance techniques that protect that journey without overengineering the whole system.

Original DosiBridge notes for Fault Tolerance

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Fault Tolerance is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• SLO: For Fault Tolerance, decide the SLO before choosing tools; otherwise the design looks clean but fails when fault changes.
• Capacity Headroom: For Fault Tolerance, decide the capacity headroom before choosing tools; otherwise the design looks clean but fails when tolerance changes.
• Failure Mode: For Fault Tolerance, decide the failure mode before choosing tools; otherwise the design looks clean but fails when fault changes.
• Degradation Path: For Fault Tolerance, decide the degradation path before choosing tools; otherwise the design looks clean but fails when tolerance changes.
• Operator Signal: For Fault Tolerance, decide the operator signal before choosing tools; otherwise the design looks clean but fails when fault changes.

Production review question

If a team adopts Fault Tolerance, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track fault-tolerance health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Fault Tolerance, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.