Design Cloud Storage (S3)

Problem Statement

Design an object storage system like AWS S3 that stores arbitrary binary objects (files, images, backups) with high durability and availability. The system must support PUT/GET/DELETE operations on objects up to 5 TB, organize objects in buckets, provide 11 nines (99.999999999%) of durability, and scale to exabytes of data across multiple data centers.

Requirements

Functional

• Create buckets; PUT/GET/DELETE objects within buckets identified by key (path-like string)
• Support objects up to 5 TB via multipart upload (upload in 100 MB parts, assemble server-side)
• Object versioning: keep all versions of an object, retrieve by version ID
• Access control: bucket policies and per-object ACLs

Non-Functional

• Durability: 99.999999999% (11 nines) -- losing data is unacceptable
• Availability: 99.99% for reads, 99.9% for writes
• Scale: Exabytes of total storage, 100M+ objects per bucket, 100K requests/second
• Latency: First byte in <100ms for small objects, <1 second for large objects

Core Architecture

1. API Gateway -- Handles authentication (HMAC-signed requests), authorization (bucket policies + IAM), request routing, and rate limiting per tenant. Parses the bucket and object key from the URL. Routes to the metadata service for lookups and the data service for actual bytes.

2. Metadata Service -- Maps (bucket, key, version) -> object metadata (size, content-type, checksum, creation time, data placement: which data nodes hold the chunks). Backed by a distributed key-value store (DynamoDB-like or Cassandra) partitioned by hash(bucket + key). The metadata store itself is replicated 3x for durability.

3. Data Service with Erasure Coding -- Objects are split into data chunks and encoded using Reed-Solomon erasure coding (e.g., 6 data + 3 parity = 9 chunks). Any 6 of the 9 chunks can reconstruct the original. Chunks are distributed across 9 different storage nodes in different failure domains (racks, zones). This provides 11-nines durability with only 1.5x storage overhead (vs. 3x for triple replication).

4. Placement Engine -- Decides which storage nodes receive each chunk. Ensures failure domain diversity: no two chunks of the same object on the same rack, power circuit, or availability zone. Uses a consistent hash ring weighted by node capacity. Rebalances chunks when nodes are added or decommissioned.

5. Garbage Collector -- Handles deleted objects and old versions. Deletion marks the object as deleted in metadata (tombstone). A background GC process reclaims storage by deleting orphaned chunks after the tombstone retention period (e.g., 30 days). Also detects and repairs degraded objects (fewer than 9 healthy chunks) by re-encoding and placing new chunks.

Database Choice

Custom distributed KV store (DynamoDB-style) for metadata -- must handle 100M+ objects per bucket with fast lookup by key. Partitioned by hash(bucket + key). Local filesystems (XFS/ext4) on storage nodes for actual chunk data -- each node manages its own disk array. No traditional database touches the data path for reads/writes (latency sensitive). PostgreSQL for bucket configuration, IAM policies, and billing records (off the hot path).

Key API Endpoints

PUT /\{bucket\}/\{key\}
  -> Body: <binary object data>
  -> Headers: Content-Type, x-amz-meta-*, Content-MD5
  -> Returns: \{ ETag: "abc123...", VersionId: "v1" \}

GET /\{bucket\}/\{key\}?versionId=v1
  -> Returns: Binary object data with Content-Type and metadata headers

POST /\{bucket\}/\{key\}?uploads (initiate multipart upload)
  -> Returns: \{ UploadId: "UP-789" \}

Scaling Insight

Erasure coding (Reed-Solomon 6+3) is the key to achieving 11-nines durability affordably. With triple replication (3 copies), you need 3x storage and can tolerate 2 simultaneous failures. With RS(6,3), you need only 1.5x storage and can still tolerate 3 simultaneous failures -- double the fault tolerance at half the storage cost. At exabyte scale, this difference saves billions of dollars in hardware. The tradeoff is CPU cost for encoding/decoding, but modern hardware can encode at 10+ GB/s per core.

Key Tradeoffs

Original DosiBridge notes for Design Cloud Storage (S3)

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Design Cloud Storage (S3) is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Core Entity: For Design Cloud Storage (S3), decide the core entity before choosing tools; otherwise the design looks clean but fails when cloud changes.
• Read/Write Path: For Design Cloud Storage (S3), decide the read/write path before choosing tools; otherwise the design looks clean but fails when storage changes.
• Contention Point: For Design Cloud Storage (S3), decide the contention point before choosing tools; otherwise the design looks clean but fails when cloud changes.
• Abuse Path: For Design Cloud Storage (S3), decide the abuse path before choosing tools; otherwise the design looks clean but fails when storage changes.
• Recovery Workflow: For Design Cloud Storage (S3), decide the recovery workflow before choosing tools; otherwise the design looks clean but fails when cloud changes.

Design Cloud Storage (S3) correctness boundary

For this problem, separate the user-visible action from background work. The correctness boundary is the smallest state transition that must not be duplicated, lost, or reordered. Naming that boundary makes the solution more original than a stock API/database/cache template.

Design Cloud Storage (S3) scaling bottleneck

The first bottleneck should be described in the language of this system: a hot timeline, a contested seat, a busy wallet ledger, a media-processing queue, or a search index update delay. That bottleneck decides which component deserves the deepest interview discussion.

Production review question

If Design Cloud Storage (S3) suddenly receives 10x traffic, which entity becomes contested first, and what queue, lock, shard, or cache protects that entity without hiding correctness bugs?

Practical metric to track

Track cloud-storage health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Design Cloud Storage (S3), this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• Design Cloud Storage -- Reference
• Source: System-Design-Overview