What Is a Data Lakehouse? Lakehouse Architecture Explained

A data lake stores anything (JSON, CSV, Parquet, images) cheaply on object storage, but it's awkward for reliable SQL analytics. A data warehouse stores structured tables optimized for fast SQL, but storage and compute cost more and it's less flexible. A lakehouse keeps data in open files on cheap object storage and adds a table layer on top, so the same data behaves like warehouse tables: ACID writes, schemas, fast queries, time travel.

See data warehouse vs data lake for the full comparison.

Why not just dump Parquet files in S3?

Early data lakes did exactly that, and hit three problems:

1. No ACID guarantees. Plain files have nothing coordinating writers. Two Spark jobs writing the same table can overwrite each other, records go missing or get corrupted, and nobody gets an error.
2. No schema enforcement. Someone writes a string into an integer column and it goes in silently; you find out when a downstream pipeline breaks.
3. The small file problem. Streaming jobs write thousands of tiny Parquet files; opening and closing each one adds I/O overhead, so queries are much slower than reading a few large files.

Open table formats (Delta Lake, Apache Iceberg, Apache Hudi) fix all three by keeping a transaction log or metadata tree alongside the files: ACID transactions with many concurrent writers, schema enforcement and evolution, time travel (query the table as it was last week), and compaction (OPTIMIZE in Delta) that merges small files into large ones. See open table formats.

The layers of a lakehouse

• Storage: Parquet files on S3, ADLS or GCS: cheap, durable, effectively unlimited. See object storage.
• Table format: Delta Lake, Iceberg or Hudi metadata that turns folders of files into tables.
• Catalog: maps table names to locations and holds schemas and permissions (AWS Glue Data Catalog, Databricks Unity Catalog, Hive Metastore, Iceberg REST catalogs).
• Engines: Spark, Databricks SQL, Trino, Athena, Snowflake, Redshift query the same tables.
• Organization: data usually flows through medallion layers: bronze (raw), silver (cleaned and modeled), gold (aggregated).

A common AWS-flavored hybrid: keep the lake in S3, register tables in the Glue Data Catalog so Athena can query S3 directly, and keep a Redshift warehouse for the most performance-critical serving tables. Redshift is the warehouse and S3 is the lake, with the catalog joining them.

Interoperability and choosing a format

Formats are open, but engines don't support every feature of every format equally. An Iceberg table written by one vendor's engine may not be read as efficiently by another's, and Delta features may only be fully supported in Databricks. Pick the format your main engines support best, and check read and write support (not just reads) for each engine you plan to use. Databricks is today's most mature end-to-end lakehouse platform; Iceberg has the broadest multi-engine adoption.

-- Delta Lake (Spark SQL): a governed lakehouse table on object storage
CREATE TABLE silver.orders (
  order_id STRING, customer_id STRING, amount DECIMAL(10,2), order_ts TIMESTAMP, order_date DATE
) USING DELTA
PARTITIONED BY (order_date);

MERGE INTO silver.orders t                -- idempotent upsert from the bronze batch
USING bronze_orders_batch s ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

SELECT * FROM silver.orders VERSION AS OF 42;   -- time travel
OPTIMIZE silver.orders;                          -- compact small files