One copy of the data, three formats: what XTable and UniForm change in the test plan

Head office runs its lakehouse on Delta Lake, under Databricks and Microsoft Fabric. A subsidiary acquired two years ago runs Snowflake and reads Iceberg. Group finance needs one margin figure, so a nightly job copies the sales tables from one side to the other. Eighteen months later there are two margins, a reconciliation spreadsheet with a named owner, and an AI assistant on each side that confidently quotes its own number.

The copy was never a data problem. It was a metadata problem, and that is what makes it solvable without moving a byte.

Why the same files can be three tables

Every open table format stores its data the same way: immutable Parquet files on object storage. What differs is the metadata beside them. Delta Lake keeps a transaction log of JSON commits, Iceberg a tree of snapshot, manifest list and manifests, Hudi a timeline. The table is the metadata; the files are shared ground. The background is in Operational databases and the lakehouse.

Translating a table from one format to another therefore means writing a second set of metadata that points at the same files: the same list of files, the same schema, the same partition values and statistics, expressed in the other format’s structure. Two projects do this.

Apache XTable, incubating at the Apache Software Foundation, reads the metadata of a source format and writes the metadata of one or more target formats beside it. Its documentation is explicit that it does so without copying or moving the underlying data files. It runs as a tool or a scheduled job, in a FULL mode that rebuilds the target metadata from all files or an INCREMENTAL mode that carries over only the difference. The current release is 0.4.0-incubating, which added a Delta Kernel target and Paimon and Parquet as sources (checked 2026-10-21). Documented limits: Merge-on-Read tables in Hudi and Iceberg and Delta tables with deletion vectors are not supported.

Delta UniForm does the same from inside Delta Lake. With the feature enabled, each commit also generates Iceberg metadata, and Hudi metadata in preview, asynchronously after the Delta commit. The Delta documentation lists what it requires: column mapping, reader version 2 and writer version 7, IcebergCompatV2, Delta 3.1 or later for Iceberg and 3.2 for Hudi. The Iceberg and Hudi views are read-only. Deletion vectors are not carried to an Iceberg v2 reader, the VOID type is unsupported and the Change Data Feed is not exposed through Iceberg; Databricks documents that REORG ... UPGRADE UNIFORM purges deletion vectors for that reason.

Both approaches leave one format as the format of record, the one that is written. The others are views of it, refreshed with a lag.

Infographic

Two ways to give a second engine the same table

Copy the data

• –A nightly job rewrites the files into the other platform
• –Two sets of files, two storage bills, two freshness clocks
• –Two tables that drift, and a reconciliation somebody owns
• –Permissions set twice, usually differently
• –Every schema change made in two places

Translate the metadata

• →One set of Parquet files, one format of record
• →A second set of metadata written beside the first (XTable) or by the writer itself (UniForm)
• →One table, read natively by both engines, with a known lag
• →Permissions still set per catalogue, which is the part to test
• →Unsupported features found by a contract test, not by a user

What changes in the test plan

A copied table is tested by reconciliation: the two sides are compared after the load. A translated table is one table seen twice, and the failure modes move. In a spec-driven delivery the test plan is part of the specification, so these are the tests we write before the first sync runs.

Cross-format equivalence. The same table read through the format of record and through the translated view: row counts, a fixed set of aggregates per partition, the schema with its types, and the latest commit each side can see. Run after every sync and on a schedule. A mismatch is a defect in the pipeline, not a reconciliation item.

Freshness lag. The translated view is behind the write by design: XTable by its schedule, UniForm by its asynchronous generation, OneLake’s virtualization in Fabric by what Microsoft documents as five seconds to two minutes for tables under five thousand commits. The contract states the lag the consumer is promised; the test measures it and fails when the promise is broken. An assistant that answers from the Iceberg view needs to know, and ideally say, how old its view is.

Feature contracts. Every documented limit becomes a test. A table that has deletion vectors enabled, a VOID column, a Merge-on-Read layout or a reliance on the Change Data Feed fails its contract test before it reaches the translation step, with a message that names the limit. These tests are cheap to write and expensive to lack.

Permission parity. Metadata translation says nothing about who may read. The Delta table under Unity Catalog and its Iceberg view under Snowflake’s or Polaris’s catalogue carry separate grants. The test enumerates the principals on both sides and fails on a difference the specification did not authorise.

Rollback of the view. If a sync produces a wrong Iceberg snapshot, the test plan says how to return the view to the previous one and how consumers learn that it happened.

Who owns the write format

Translation removes the copy and keeps a decision: which format is written. That decision binds the engines that can write, the catalogue that governs, and the features the platform can use without breaking the view. It is an architecture decision with a named owner, not a configuration flag.

The market context matters here only as fact. Databricks announced the acquisition of Tabular in June 2024 with compatibility between Delta and Iceberg through UniForm as the stated short-term goal. Microsoft documents OneLake’s virtualization of Delta tables as Iceberg and Iceberg shortcuts. Confluent’s Tableflow materialises Kafka topics as Iceberg or Delta tables. The direction is that platforms read each other’s formats; the ownership of the written format, and of the catalogue, has not moved.

Where agents help, and where an architect decides

The test suites above are repetitive, parameterised and numerous, which is exactly what coding agents produce well from a specification: the equivalence queries per table, the lag monitors, the contract checks generated from the documented limits, the permission diff. An agent can also read a failed sync’s log and name the feature that broke it. We let agents draft all of that and accept it by running it against real tables.

The decisions stay with people: which format is the format of record; whether a table may enable a feature the view cannot carry; what lag each consumer is promised; whether a subsidiary’s platform is a view or a second writer; and when a translated view is good enough to switch off the copy. The switch-off is the moment the whole exercise pays, and it should happen only after the equivalence tests have passed for long enough that nobody is reading the reconciliation spreadsheet any more.

What a good outcome looks like

One set of files for the group’s sales tables, written in one format by one platform. The subsidiary’s engine reads them natively, with a lag that is stated and measured. The nightly copy and its reconciliation are gone. Every limit of the translation is a test that runs before the user notices. Both assistants quote the same margin, and when they are asked how fresh it is, they answer.

Our part

In AI agents spec-driven development we write the specification first, the test plan with it, and let agents generate the equivalence, lag and contract suites that make a translated table trustworthy, accepted against your real tables before a copy is switched off. We do not resell platforms. More about predictes is on the home page.