intermediate10 min readUpdated 2026-06-08

Data Lakes: Raw Storage, Governance and Analytics Architecture

Learn data lake architecture: raw object storage, zones, catalogs, governance, file formats, query engines, lakehouse patterns, and tradeoffs.

Quick answer: Data Lakes: Raw Storage, Governance and Analytics Architecture

Data Lakes: Raw Storage, Governance and Analytics Architecture is covered here as a practical system design concept. Learn data lake architecture: raw object storage, zones, catalogs, governance, file formats, query engines, lakehouse patterns, and tradeoffs. Use it to understand the Databases idea, where it appears in production systems, and how to discuss it clearly in interviews.

Focus on the problem being solved, the main design choice, the failure mode, and the tradeoff you would defend in front of a senior engineer. This page is part of the DosiBridge system design knowledge graph, so it links related concepts, interview problems, tradeoffs, and references instead of leaving the topic isolated.

Aspect	Details
What it is	A centralized storage repository for raw data in any format, using cheap object storage and schema-on-read to defer data structuring until query time
When to use	When you need to store diverse data types (logs, JSON, images, CSVs) cheaply, when schema requirements are uncertain, or when multiple teams need access to raw data for different analytical purposes
When NOT to use	When you need fast, structured analytical queries with guaranteed schema — use a data warehouse instead; or when data governance and quality are critical from day one
Real-world example	Netflix stores all raw event data (billions of streaming events daily) in their S3-based data lake, which feeds downstream data warehouses, ML pipelines, and analytics systems
Interview tip	Always contrast with data warehouses — lakes store raw data cheaply with flexibility but risk becoming 'data swamps'; warehouses store clean data with governance but are more expensive and rigid
Common mistake	Dumping data into a lake without any organization, cataloging, or governance — creating a 'data swamp' where nobody can find or trust the data
Key tradeoff	Data lakes offer maximum storage flexibility and low cost but require significant effort in governance, cataloging, and access control to remain useful

Why This Matters

Data lakes emerged because traditional data warehouses couldn't handle the volume, variety, and velocity of modern data. Storing petabytes of logs, sensor data, clickstreams, and unstructured content in a warehouse would be prohibitively expensive and require predefined schemas for every data source. Data lakes solve this by storing everything in cheap object storage (S3, ADLS) at $0.02/GB/month — 10-50x cheaper than warehouse storage. The schema-on-read approach means data scientists and engineers can explore raw data without waiting for schema design and ETL pipeline development. However, without governance, data lakes become data swamps — understanding when to use a lake vs a warehouse is a key architectural skill.

System architecture diagram for Data Lakes showing how services, databases, and caches connect — System architecture for Data Lakes

The Building Blocks

Object Storage Layer: The foundation is cheap, durable object storage (S3, ADLS, GCS) that handles petabytes of data with 99.999999999% durability and pay-per-use pricing.
Schema-on-Read: Data is stored raw without predefined schema. Structure is applied at query time using tools like Apache Spark, Presto, or Athena, allowing different consumers to interpret the same data differently.
File Formats: Columnar formats like Parquet and ORC provide compression and column pruning for analytics; Avro provides row-based storage for streaming; Delta Lake and Iceberg add ACID transactions.
Data Catalog: A metadata catalog (AWS Glue Catalog, Apache Hive Metastore) registers datasets with schemas, locations, and ownership so users can discover and understand available data.
Table Formats: Modern table formats like Delta Lake, Apache Iceberg, and Apache Hudi add warehouse-like features to lakes — ACID transactions, schema evolution, time travel, and incremental processing.

Under the Hood

A data lake is fundamentally an object storage system organized into zones. The raw zone (or bronze layer) contains data exactly as ingested — no transformation, no deduplication. This is the immutable source of truth. The cleaned zone (silver layer) contains validated, deduplicated, and standardized data. The curated zone (gold layer) contains business-level aggregations and models ready for consumption. This multi-zone approach, called the medallion architecture, provides both flexibility and structure.

Step-by-step diagram showing how Data Lakes processes a request from start to finish — How Data Lakes works step by step

Data arrives via batch ingestion (scheduled ETL jobs pulling from databases and APIs), streaming ingestion (Kafka or Kinesis writing events directly to storage), or file drops (partners uploading CSVs to designated S3 prefixes). Each dataset is registered in a data catalog that stores its location, schema, owner, and freshness metadata. Users query the lake through engines like Apache Spark, Presto/Trino, or serverless services like AWS Athena and Azure Synapse Serverless.

The evolution from basic data lakes to "lakehouses" addresses the lake's original weaknesses. Delta Lake (created by Databricks), Apache Iceberg (created by Netflix), and Apache Hudi (created by Uber) add transaction logs on top of object storage, enabling ACID transactions, schema evolution, partition evolution, time travel (querying data as of a past timestamp), and efficient upserts. These table formats effectively merge the flexibility of data lakes with the reliability of data warehouses, which is why the "lakehouse" architecture has become the dominant modern pattern.

How Companies Actually Do This

Netflix Stores all raw event data from 200+ million subscribers in an S3-based data lake, organized in the medallion architecture. Apache Iceberg provides ACID transactions and time travel for their petabyte-scale analytical workflows.

Comparison table for Data Lakes contrasting approaches, tradeoffs, and when to use each — Comparing key aspects of Data Lakes

Uber Built a massive data lake on HDFS and S3, using Apache Hudi (which they created) to enable incremental processing and upserts on petabyte-scale datasets for real-time analytics on ride data, driver behavior, and marketplace dynamics.

Databricks Created Delta Lake to add ACID transactions to cloud object storage, enabling their lakehouse platform that combines the cost and flexibility of data lakes with the performance and governance of data warehouses for thousands of enterprise customers.

Common Pitfalls

Creating a data swamp — dumping data without organization, cataloging, or ownership leads to a lake where nobody can find anything and nobody trusts the data quality
Not using columnar file formats — storing analytics data as CSV or JSON instead of Parquet or ORC wastes 5-10x more storage and makes queries 10-100x slower due to lack of compression and column pruning
Ignoring access control — data lakes often contain sensitive data from many sources; without fine-grained access control (row-level, column-level), you risk exposing PII or violating compliance regulations

Data flow diagram for Data Lakes showing how requests and responses move through the system — Data flow through Data Lakes

Interview Questions Worth Practicing

How would you design a data lake architecture that prevents it from becoming a data swamp?
Compare data lakes and data warehouses — when would you use each, and how do lakehouses combine both?
Explain the medallion architecture (bronze/silver/gold layers) and why each layer exists.

The Tradeoffs

Flexibility vs Governance: Data lakes accept any data format without upfront schema design, but this flexibility makes governance, quality, and discoverability harder to enforce
Cost vs Query Performance: Object storage is 10-50x cheaper than warehouse storage, but query performance on raw files is significantly slower without proper partitioning, formats, and table formats
Schema-on-Read vs Schema-on-Write: Deferring schema design to query time maximizes ingestion speed and flexibility but pushes complexity to consumers who must understand raw data structures

Component diagram for Data Lakes showing each building block and its responsibility — Key components of Data Lakes

How to Explain This in an Interview

Here is how I would explain Data Lakes in a system design interview:

Start by defining a data lake as centralized storage for raw data in any format — structured, semi-structured, unstructured — using cheap object storage like S3 at $0.02/GB/month. Contrast with data warehouses: warehouses use schema-on-write (data must be structured before loading), while lakes use schema-on-read (structure applied at query time). Explain the medallion architecture: bronze layer (raw data as ingested), silver layer (cleaned and validated), gold layer (business-ready aggregations). Address the 'data swamp' risk — without a data catalog, ownership policies, and quality checks, lakes become unusable. Then discuss lakehouses: modern table formats like Delta Lake, Iceberg, and Hudi add ACID transactions, schema evolution, and time travel to lake storage, merging lake flexibility with warehouse reliability. This lakehouse pattern is the modern best practice, and it's worth mentioning that major cloud providers now offer lakehouse-native services.

Interview preparation checklist for Data Lakes with key points to mention and mistakes to avoid — Interview tips for Data Lakes

Original DosiBridge notes for Data Lakes: Raw Storage, Governance and Analytics Architecture

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Data Lakes: Raw Storage, Governance and Analytics Architecture is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

Query Shape: For Data Lakes: Raw Storage, Governance and Analytics Architecture, decide the query shape before choosing tools; otherwise the design looks clean but fails when data changes.
Write Amplification: For Data Lakes: Raw Storage, Governance and Analytics Architecture, decide the write amplification before choosing tools; otherwise the design looks clean but fails when lakes changes.
Replication Lag: For Data Lakes: Raw Storage, Governance and Analytics Architecture, decide the replication lag before choosing tools; otherwise the design looks clean but fails when data changes.
Migration Safety: For Data Lakes: Raw Storage, Governance and Analytics Architecture, decide the migration safety before choosing tools; otherwise the design looks clean but fails when lakes changes.
Data Ownership: For Data Lakes: Raw Storage, Governance and Analytics Architecture, decide the data ownership before choosing tools; otherwise the design looks clean but fails when data changes.

Production review question

If a team adopts Data Lakes: Raw Storage, Governance and Analytics Architecture, what observable signal proves it helped instead of only making the architecture diagram more fashionable?

Practical metric to track

Track data-lakes health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Data Lakes: Raw Storage, Governance and Analytics Architecture, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.