Disaster Recovery
Disasters happen: AWS us-east-1 has had multi-hour outages, entire data centers have lost power, and ransomware attacks have encrypted production.
Quick answer: Disaster Recovery
Disaster Recovery is covered here as a practical system design concept. Disasters happen: AWS us-east-1 has had multi-hour outages, entire data centers have lost power, and ransomware attacks have encrypted production. Use it to understand the Distributed Systems idea, where it appears in production systems, and how to discuss it clearly in interviews.
Focus on the problem being solved, the main design choice, the failure mode, and the tradeoff you would defend in front of a senior engineer. This page is part of the DosiBridge system design knowledge graph, so it links related concepts, interview problems, tradeoffs, and references instead of leaving the topic isolated.
The Problem Disaster Recovery Solves
Disasters happen: AWS us-east-1 has had multi-hour outages, entire data centers have lost power, and ransomware attacks have encrypted production databases. Without DR, these events cause extended downtime and permanent data loss.
How It Works Under the Hood
Disaster recovery (DR) is the set of policies, tools, and procedures to recover from catastrophic failures — data center outages, region failures, natural disasters, or large-scale cyberattacks. DR ensures business continuity by maintaining backup systems in geographically separate locations.
A typical DR setup: Primary region (US-East) handles all traffic. Data is continuously replicated to a secondary region (US-West). Monitoring detects a primary region failure. DNS or global load balancer switches traffic to the secondary region. Users experience a brief disruption but service continues.
For active-active: both regions handle traffic simultaneously. A region failure simply means remaining regions handle all traffic (at higher utilization).
The Mental Model
- RPO (Recovery Point Objective): Maximum acceptable data loss measured in time. RPO of 1 hour means you can lose at most 1 hour of data.
- RTO (Recovery Time Objective): Maximum acceptable downtime. RTO of 4 hours means the system must be back online within 4 hours.
- Active-passive DR: A standby environment in a different region receives replicated data. On disaster, traffic is switched to the standby.
- Active-active DR: Both regions serve traffic simultaneously. A disaster in one region simply reduces capacity.
- Backup tiers: Hot (always running, seconds to failover), Warm (running but not serving, minutes to failover), Cold (data only, hours to restore).
Real Systems That Depend on This
Netflix runs active-active across 3 AWS regions. Traffic is continuously routed to all regions. A region failure is absorbed by the other two.
GitHub maintains a warm standby in a separate data center with continuously replicated data.
Google Spanner provides active-active multi-region replication with strong consistency using TrueTime.
Where This Shows Up in Interviews
- What is RPO and RTO?
- How do you design a system for disaster recovery?
- What is the difference between active-passive and active-active DR?
- How do you test disaster recovery?
Tradeoffs
- Cost vs Recovery speed: Active-active is fast but expensive (2x infrastructure). Cold standby is cheap but slow.
- Consistency vs Availability: Synchronous cross-region replication guarantees zero data loss but adds latency.
- Complexity: Multi-region architectures are significantly more complex to build, test, and operate.
Watch Out For
- Not testing DR procedures — they do not work when you need them
- RPO/RTO not aligned with business requirements — over-engineering wastes money, under-engineering causes data loss
- Not accounting for DNS propagation time during failover
Go Deeper
- Availability — start here if this is new to you
- fault-tolerance
- failover
- Data Replication
Original DosiBridge notes for Disaster Recovery
This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Disaster Recovery is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.
What makes this topic distinct
- Membership: For Disaster Recovery, decide the membership before choosing tools; otherwise the design looks clean but fails when disaster changes.
- Clock Drift: For Disaster Recovery, decide the clock drift before choosing tools; otherwise the design looks clean but fails when recovery changes.
- Retry Semantics: For Disaster Recovery, decide the retry semantics before choosing tools; otherwise the design looks clean but fails when disaster changes.
- Partition Behavior: For Disaster Recovery, decide the partition behavior before choosing tools; otherwise the design looks clean but fails when recovery changes.
- Reconciliation: For Disaster Recovery, decide the reconciliation before choosing tools; otherwise the design looks clean but fails when disaster changes.
Production review question
If a team adopts Disaster Recovery, what observable signal proves it helped instead of only making the architecture diagram more fashionable?
Practical metric to track
Track disaster-recovery health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Disaster Recovery, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.