Delta Lake: the transaction log under Fabric and Databricks, and what you still own

A manufacturer on Microsoft 365 switches on Fabric, points a pipeline at the ERP and the MES, and within a week has tables that Power BI reads and a Copilot answers from. Nobody in the company has said the words “Delta Lake”. Six months later a works council asks for the deletion of a former employee’s records from every system, and the data team discovers that the lakehouse keeps every version of every table for as long as nobody tells it otherwise.

Delta Lake is the table format under OneLake in Microsoft Fabric, which stores tables “in the open Delta Parquet format”, and under Databricks. For many enterprises it is the first table format they meet, usually without choosing it. That is a reasonable way to start. It is not a reason to leave the parts that are still yours undecided. In the pillar on the two memories of an AI-native system the lakehouse is the second memory; Delta is one way of keeping that memory consistent, and this article is about where the platform’s responsibility ends.

What the log does

A Delta table is a directory of Parquet files plus _delta_log/. Each commit is a JSON file named with a twenty-digit, zero-padded version number and containing actions: add and remove for data files, metaData for the schema and partitioning, protocol for the minimum reader and writer versions, commitInfo, txn, cdc and domainMetadata. Checkpoints in Parquet — V2 checkpoints with sidecar files in current releases — summarise the state, and _last_checkpoint tells a reader where to start. The protocol document is the contract; every engine that reads or writes Delta implements it, directly or through Delta Kernel, the Java and Rust libraries built so that a connector does not have to.

From the log follow the properties the platform advertises: atomic commits, so a failed load leaves the table as it was; isolation between readers and writers; time travel to any retained version; MERGE, UPDATE and DELETE without a database underneath; one table for batch and stream.

The current release is Delta Lake 4.4.0, checked 2026-10-21. Behaviour beyond the base protocol is gated by table features, each with a minimum reader and writer version and a release in which it arrived:

Feature	What it changes	Reader / writer
Deletion vectors (2.3.0)	A delete marks rows instead of rewriting files	3 / 7
Column mapping (1.2.0)	Rename or drop a column without rewriting data	2 / 5
Change Data Feed (2.0.0)	A table emits its own row-level changes	1 / 4
Row tracking (3.2.0)	Stable row identities across rewrites	3 / 7
Liquid clustering (3.1.0)	Data layout by clustering keys instead of fixed partitions	—
Type widening (GA 4.0.0)	Widen a column type in place	3 / 7
Variant type (4.0.0)	Semi-structured data as a native type	3 / 7
UniForm (IcebergCompatV2, 3.1.0)	Iceberg metadata generated beside the log	—

Source: Delta versioning, checked 2026-10-21. The column on the right is the one to read before enabling anything: a feature that raises the reader version is invisible to any engine that has not caught up.

What the platform runs for you, and what it does not

On Fabric and Databricks the log is written by the platform’s engines, the storage is managed, and the catalogue — OneLake’s or Unity Catalog — governs who sees which table. Optimisation jobs can be scheduled from the platform. Reading Delta from outside is possible through the open protocol, and reading it as Iceberg through UniForm. That is a lot, and it is why the format rarely needs to be chosen in a Microsoft estate.

What the platform does not do is decide. Four decisions stay with the enterprise however managed the platform is.

Infographic

What stays yours under a managed Delta platform

• The catalogue and its permissionsWho may read which table, from which engine, under which identity. The format carries none of this.
• Retention, VACUUM and deletionsA deleted row is gone from the current version at once and from the files only after VACUUM. Until then it is still on disk and still in time travel.
• Feature enablementEach table feature raises the minimum reader or writer. Enabling one for convenience can cut off a team reading from another engine.
• Service jobs and their costOPTIMIZE, clustering, checkpointing and VACUUM run on your compute, on your schedule, with somebody watching.
• The write pathWhich pipeline is allowed to write to a table, and whether a coding agent may change it without review.

The retention point deserves the detail. Delta keeps the log for 30 days by default and VACUUM removes files older than 7 days by default (utilities, checked 2026-10-21). A DELETE writes a new version without the rows — with deletion vectors, it marks them — and the old files stay until VACUUM removes them. A deletion request under GDPR is therefore two steps, the second on a schedule you own, and an auditor may ask for the date of the second. Time travel to a version before the deletion must stop working at that point, which is the price of the deletion being real.

A teaching example: ERP and MES into Fabric

Take a manufacturer with an ERP for orders and stock and an MES for production events, both landing in a Fabric lakehouse. This is an illustration of the design, not a client account.

The MES emits machine states and completed operations continuously; the ERP changes orders, routings and stock levels during the day. Both land through CDC into Delta tables. A coding agent drafts the MERGE pipelines from the source schemas and the keys, drafts the OPTIMIZE and clustering schedule from the observed write pattern, and enables Change Data Feed on the order table so the planning assistant downstream reads only what changed. All of that goes through a pull request with the test results: row counts against the source, a MERGE replayed twice with the same result, a time-travel read to yesterday’s version.

What the engineer decides, and records next to the table: that production events are retained for the warranty period and not longer; that VACUUM runs weekly and the deletion register is checked against it; that deletion vectors are enabled because the engines in the estate all read protocol 7, with the reader versions written down; that the Snowflake subsidiary will read these tables through UniForm and therefore column mapping is enabled from the start, because UniForm requires it. The planning assistant reads the lakehouse and the semantic layer; it writes nothing back to the ERP except through the ERP’s own interface, as a proposed change that a planner confirms.

How we approach it

We start from the deletion request and the audit question, not from the pipeline, because they decide retention, and retention decides the rest. The first table goes through the whole path: CDC in, MERGE, feature decisions with reader versions noted, service jobs scheduled and costed, one reader outside the platform tested. Coding agents draft the pipelines and the tests; the pull request carries the reasoning; a named engineer approves. The result is a Delta estate where the platform does what it is good at and the enterprise knows exactly which four things are still its own.

Our part

In AI data engineering we build the Delta path under a Fabric or Databricks platform all the way through — ingestion, feature decisions, retention and service jobs, with the outside readers tested — and leave you a table whose decisions are written down. We do not resell platforms or licences. More about predictes is on the home page.