Open Table Formats: Apache Iceberg, Delta Lake and Hudi

Open table formats such as Apache Iceberg, Delta Lake and Apache Hudi turn a folder of data files on object storage into a real table. They track exactly which files belong to each version of the table in metadata, so writes become atomic commits, readers always see a consistent snapshot, schemas can evolve safely, and you can query the table as it was yesterday. They're what makes a lakehouse possible.

The problem with plain files

• No atomicity: a job writing 500 files fails after 300; readers now see a partial table.
• No isolation: a reader listing files during a write sees some old and some new files.
• Slow planning: listing millions of files on object storage just to plan a query is slow and costly.
• No safe updates or deletes: changing one row means rewriting files with no coordination, which matters for corrections and privacy deletion requests.
• Fragile schema changes: renaming or reordering a column can silently misalign old files.

How a table format works

The table's state is a chain of snapshots described by metadata files. Each snapshot lists the data files that make up the table at that moment, with per-file statistics. A writer creates new data files, writes new metadata describing the next snapshot, then atomically swaps the catalog's pointer to that metadata. If two writers race, one commit wins and the other retries on top of it (optimistic concurrency).

Delta Lake records the same information as an ordered transaction log of JSON commits with periodic checkpoints. Hudi uses a timeline of actions. The mechanisms differ, but all three give atomic commits and snapshot reads.

What you get

Copy-on-write vs merge-on-read

Operating them

• Compaction: rewrite many small files into fewer large ones (see the small files problem), and fold merge-on-read deltas into base files.
• Snapshot expiry: old snapshots keep old files alive; expire them on a schedule, which also sets how far back time travel works.
• Orphan file cleanup: remove files left behind by failed writes.
• Catalog choice: the catalog (Hive metastore, a REST catalog, or a managed one) is the source of truth for the current table version and the place to apply access control.
• Deletion requirements: a DELETE removes rows from the current snapshot, but data stays in old files until snapshots expire and files are cleaned up.

Choose by ecosystem more than features: which engines and catalogs your team already uses, and which format they support best. The core guarantees are similar.