Apache Iceberg: the table format your exit plan can rely on

A group acquires a company and inherits its data estate: the group runs Trino over its own object storage, the acquired company runs Snowflake, and both now need the same customer and order tables. The usual answer is a nightly copy in each direction and two teams who stop trusting each other’s numbers by spring. The less usual answer is one set of tables that both engines read and write, which is what Apache Iceberg was designed for.

Iceberg is the table format on which the most vendors have agreed: Snowflake, AWS, Google Cloud, Databricks, Microsoft Fabric through OneLake, and the open engines — Spark, Flink, Trino, DuckDB. The agreement comes from the architecture, not from goodwill, and the architecture is also where the remaining dependency sits. In the pillar on the two memories of an AI-native system the lakehouse is the memory that outlives any one engine; Iceberg is the format that takes that promise most literally, provided the catalogue is treated as the decision it is.

The tree, and the one thing that moves

An Iceberg table is a tree of immutable files. The specification, checked 2026-10-21, lays it out: a table metadata file holds the schema, the partition specs and the list of snapshots; each snapshot points to a manifest list; the manifest list points to manifests; each manifest, an Avro file, lists data files and delete files with column statistics. A query plans by walking down this tree and pruning on the statistics before it opens a single Parquet file.

Nothing in the tree is ever modified. A commit writes new manifests, a new manifest list and a new metadata file, then asks the catalogue to swap the table’s pointer from the old metadata file to the new one, atomically, and only if nobody else swapped it first. That single swap is the only mutable operation. It is why several engines from different vendors can write to one table without coordinating with each other: they coordinate with the catalogue.

Infographic

The Iceberg metadata tree

1. 01Catalogue pointerThe current metadata file for each table. The only thing that changes in place, and the only place access is controlled.
2. 02Table metadata fileSchema, partition specs, sort orders, the list of snapshots. Immutable; a new one per commit.
3. 03SnapshotOne consistent state of the table at a point in time. Time travel reads an older one.
4. 04Manifest listWhich manifests belong to this snapshot, with partition summaries for pruning.
5. 05ManifestsAvro files listing data files and delete files with column statistics.
6. 06Data and delete filesParquet (or ORC, Avro) data; delete files and, in v3, deletion vectors in Puffin files.

Two consequences follow for an enterprise. First, the table is not the engine’s: Spark can write a snapshot that Trino reads a second later, and Snowflake can read the same snapshot from its own service. Second, whoever holds the pointer holds the table. The files may sit in your bucket in an open format, but if the catalogue is a service you cannot export or re-point, the exit plan is not finished.

What is in the current specification

Spec version 3 is adopted. It adds binary deletion vectors stored in Puffin files, row lineage, the variant type for semi-structured data, geometry and geography types, nanosecond timestamps, default column values and multi-argument transforms. Version 4 is under active development and not adopted. The library release is Iceberg 1.12.0 of 30 September 2026, which brings variant read and write to Spark, Flink and Kafka Connect and drops Spark 3.4 and Flink 2.0 (checked 2026-10-21). An engine that “supports Iceberg” may support v2 and not v3; the version is a line in the architecture document, not an assumption.

Two features matter more in enterprise use than their names suggest.

Hidden partitioning. The table declares that it is partitioned by, say, the month of the order date, as a transform of a column. Readers filter on the column and the planner converts the filter into partition pruning; nobody writes WHERE month = '2026-09' by hand or gets it wrong. A coding agent generating queries against the table does not need to know the partition layout, which removes a whole class of slow or incorrect generated SQL.

Partition evolution. When monthly partitions become too coarse, the spec is changed to daily for new data, with a new spec id. Old data stays as it is; queries plan across both. The table is not rewritten. In a five-year estate this is the difference between a change request and a migration project.

Catalogues: where the real decision sits

The REST catalogue specification defines how a client lists namespaces and tables, commits, plans scans and obtains vended credentials for the storage. Any catalogue that implements it looks the same to Spark, Trino or PyIceberg. The open implementations include Apache Polaris, a top-level Apache project since 19 February 2026 with release 1.8.0 on 28 September 2026, and Lakekeeper; the managed ones include Snowflake Open Catalog (managed Polaris), the Google Cloud lakehouse catalogue with its REST endpoint, AWS S3 Tables with Iceberg built in, and Databricks, which exposes managed Iceberg tables through a REST endpoint in Unity Catalog (vendor documentation, checked 2026-10-21).

The choice between them is the choice of who can grant access to the pointer, who can export the catalogue’s state, and what the exit looks like. An estate with Iceberg files in its own storage and a catalogue it can run or re-point has an exit plan. An estate with Iceberg files and a catalogue it can only use has half of one.

A teaching example: one table for Trino and Snowflake

Consider the group from the opening paragraph; this is an illustration of the design, not a client account. The customer and order tables are built once, in Iceberg, in the group’s object storage. A REST catalogue the group runs — Polaris or Lakekeeper — holds the pointers. Trino reads and writes through it; Snowflake reads the same tables through a catalogue integration; the Spark jobs that prepare features for the churn model read the same snapshots.

A coding agent drafts the pipelines from the source schemas, drafts the compaction and manifest rewrite jobs from the write pattern, generates the partition evolution statement when daily granularity is needed, and writes the contract tests: the same counts and sums through Trino and Snowflake, a schema change made from Spark and visible from both, a time-travel read to the snapshot the model was trained on. Each goes through a pull request with the results attached.

The engineer decides, and records: which catalogue, and that it can be exported; who holds which permission on which namespace; the snapshot expiry window, because expiring snapshots is what makes a deletion final and makes time travel stop; which spec version the estate commits to, with the readers that limit it; and that the write path to the order table belongs to one pipeline only. The churn model’s agent reads the lakehouse; it writes a proposed retention offer into the CRM through the CRM’s own interface, where a person confirms it.

How we approach it

The first table decides the estate. We build it in Iceberg with a catalogue you can run or re-point, read it from every engine that will need it, write to it from the ones that must, and leave the contract tests running in the pipeline. The compaction, snapshot expiry and manifest jobs are scheduled and costed before the second table exists. Coding agents draft the work; your architect owns the catalogue, the permissions, the retention and the spec version, with each written down.

Our part

In AI data engineering we build the Iceberg layer that an exit plan can rely on: tables in your storage, a catalogue you control, the engines you have reading and writing them, and the service jobs with an owner. We do not resell platforms or catalogues. More about predictes is on the home page.