Delta Lake, Iceberg or Hudi: choose by catalogue, not by feature list

An architecture review for a group with four plants and two ERPs reaches the slide that says “table format”. Three columns, forty rows of ticks: ACID, time travel, schema evolution, MERGE, streaming. Almost every cell is green in every column. The meeting spends an hour on the three cells that are not, and nobody asks the two questions that will actually decide the next five years: which catalogue will hold the pointer to these tables, and which engines have to write to them.

Delta Lake, Apache Iceberg and Apache Hudi have converged on the same promises. They differ in how they keep the plan of the table, and that difference shows up in the catalogue, in the engines and in the operating cost long after the feature list is forgotten. This is the second memory of an AI-native system described in the pillar on operational databases and the lakehouse; here we go one level down, to how each format keeps that memory honest.

Three ways to keep the plan of a table

All three leave the data in immutable Parquet files and describe which files make up the table at a given moment. Where they keep that description is the whole difference.

Delta Lake keeps a log. A _delta_log/ directory holds numbered JSON commits, each a list of actions: add, remove, metaData, protocol, commitInfo, cdc. Periodic Parquet checkpoints summarise the log so a reader does not replay it from the start. A writer commits by writing the next numbered file; two writers racing for the same number resolve the conflict optimistically. The current release is Delta Lake 4.4.0, checked 2026-10-21, and the protocol gates newer behaviour behind table features that raise the minimum reader and writer versions.

Iceberg keeps a tree. A table metadata file points to the current snapshot; a snapshot points to a manifest list; each manifest, itself an Avro file, lists data files and delete files with their statistics. Nothing in the tree is ever edited. A commit writes a new metadata file and asks the catalogue to swap the pointer atomically. That one swap is the only mutable operation in the whole design, which is why many engines can write safely to the same table and why the catalogue matters so much. The specification is at version 3 — deletion vectors, row lineage, the variant type, default column values — with version 4 under development and not adopted. The library release is 1.12.0, checked 2026-10-21.

Hudi keeps a timeline. Every action — commit, delta commit, compaction, clustering, clean, rollback — is an instant on the timeline with a requested, in-flight and completed state; in the 1.x line the archived history sits in an LSM structure. Beside it is a metadata table with indexes: Bloom filters, a record-level index, secondary and expression indexes. Hudi offers two table types: Copy-on-Write rewrites base files on change, Merge-on-Read appends log files and compacts later. Release 1.2.1, checked 2026-10-21, and non-blocking concurrency control lets several writers update a Merge-on-Read table with locks only on timeline metadata.

The log, the tree and the timeline are not better or worse. The log is simplest to reason about and is what Databricks and Microsoft Fabric build on. The tree separates the table from any one engine and makes the catalogue the point of control. The timeline is built for tables whose rows keep changing.

The decision table

Choose by the estate you have, not by the format you like.

Your situation	What decides	Usual answer
Databricks or Microsoft Fabric is the platform; Spark and Fabric engines do the writing	OneLake and Unity Catalog store and govern Delta natively; everything else reads Delta through UniForm or XTable	Delta Lake
Several engines (Trino, Spark, DuckDB), Snowflake or BigQuery in the estate, more than one cloud, an exit plan written into the architecture	The REST catalogue specification, Apache Polaris, S3 Tables and BigLake all speak Iceberg; writers from different vendors can share a table	Apache Iceberg
Rows change continuously — CDC from the ERP, deduplication by key, updates in minutes — and Spark or Flink runs the pipelines	Record-level index, Merge-on-Read, incremental queries and the Hudi Streamer are built for this	Apache Hudi
A single warehouse engine, nightly loads, nobody outside it needs the files	The table format is an implementation detail of the platform	Whatever the platform defaults to, documented as a decision

The rows are not exclusive. A Fabric estate with a Snowflake subsidiary is a Delta decision with an Iceberg reader, which is what XTable and UniForm are for. The acquisition of Tabular by Databricks, announced on 4 June 2024 with the stated aim of Delta and Iceberg compatibility through UniForm, is the clearest public sign that the two largest formats are converging rather than competing. It is not a reason to stop deciding.

Infographic

Where the table format is a real decision

Continuous updatesAppend-mostly

Platform defaultTake what the platform gives, write the choice down, keep an exit path.

Catalogue decidesIceberg with a catalogue you control; Delta with UniForm if the estate is Microsoft or Databricks.

Update path decidesHudi or Delta with deletion vectors; the question is who runs compaction.

Both decideMulti-engine and high churn: catalogue, concurrency model and service jobs all need an owner.

One vendor estateMany engines and clouds

What a coding agent does well with each, and where it must stop

Most pipeline code in these projects is now drafted by coding agents. That is fine for the parts that are mechanical and testable, and dangerous for the parts that are decisions dressed as configuration.

Agents draft well: a MERGE from a CDC feed into a Delta or Iceberg table with the right keys; an OPTIMIZE or compaction schedule from a description of the write pattern; catalogue configuration for a REST endpoint; the Hudi Streamer configuration from a Kafka topic and a schema registry; the contract tests that read a table through two engines and compare row counts, aggregates and the latest snapshot timestamp. Given the format’s documentation and a sample of real files, this work is faster and usually more complete than a person’s first attempt.

Agents must not decide: retention and VACUUM windows, because they are the difference between a time-travel audit that works and one that does not, and between a GDPR deletion that is final and one that is not; deletion requests themselves; protocol or format-version upgrades, because enabling a table feature in Delta or moving to Iceberg v3 can lock out an older reader in another team; the choice of catalogue, because it is the choice of who grants access; the table type in Hudi, because Copy-on-Write and Merge-on-Read move cost between writers, readers and the compaction job.

The working rule in our engagements: an agent may propose any of these, in a pull request with the reasoning and the test results attached, and a named engineer approves. The approval is recorded next to the table, not in a chat.

Contract tests before adoption

Before a format is adopted for an estate, one table is built end to end and read from every engine that will need it, with tests that run in the pipeline rather than once on a laptop. The tests are simple and the agent can write them: the same count and the same sums through engine A and engine B; a schema change (add a column, widen a type) performed from one engine and read from the other; a delete performed from one engine and invisible from the other after the same snapshot; a time-travel read to a snapshot before the delete. Where a test fails, the format matrix in the vendor documentation is consulted for the version in use — “supports Iceberg” often means version 2 without deletion vectors, and that is a fact to record, not a surprise to discover in production.

What a good outcome looks like

The architecture document names the format, the catalogue, the engines allowed to write and the engines allowed to read, with versions. The service jobs — compaction, clustering, snapshot expiry, manifest rewriting — have an owner, a schedule and a cost line. There is one table that every engine in the estate has read and written in a test, and the test still runs. Nobody is asked to choose a format from a list of ticks again, because the decision is written down with the reasons that will still hold when the list changes.

Our part

In AI data engineering we build the first table all the way through — format, catalogue, writers, readers, service jobs and the contract tests — so the format decision is made on your estate rather than on a slide. The pipelines are drafted with coding agents and accepted through tests; the decisions stay with your architect. We do not resell platforms. More about predictes is on the home page.