Apache Hudi: when the lakehouse has to keep up with the database

A distributor’s finance team reloads the receivables table every night: drop the partition for the month, write it again from the ERP, four hours of cluster time. Then sales asks for the customer’s credit position as of now, because the customer is on the phone. The nightly table is eighteen hours old, and the only current answer is in the ERP, which nobody wants the sales desk querying directly.

Apache Hudi was built for exactly this gap. Its question is not how to store a table in a lake but how to change one row in a table of a billion without rewriting the partition, and how to tell a downstream job which rows changed since it last ran. In the architecture of an AI-native enterprise system that makes Hudi the table format for the part of the second memory that has to stay close to the first.

What Hudi does that the nightly reload cannot

Three mechanisms carry the design. All of them are documented in the project’s own pages; the current release is Hudi 1.2.1 of 25 September 2026, with table version 9 and the Trino connector now maintained in the project repository (checked 21 October 2026).

A record key and an index. Every row has a key, and Hudi keeps an index in its metadata table that says which file group holds the current version of that key. The index family includes a Bloom index per file, a record-level index (global or per partition), and since release 1.0.0 secondary indexes, expression indexes and partition statistics. On the writer side the default strategy is SIMPLE, with BUCKET, HBASE and FLINK_STATE as alternatives. The index is what turns “update customer 4471” into a write to one file group instead of a scan of the month.

A timeline. Every action on the table — commit, delta commit, compaction, clustering, clean, rollback, savepoint, restore — is an instant on the timeline, with three states: requested, in flight, completed. The active timeline lives under .hoodie/timeline; the history is archived in a log-structured merge structure of Parquet files with manifests so a long-lived table does not choke on its own past. Instants are ordered by completion time. The timeline is to Hudi what the transaction log is to Delta Lake: the single source of truth about which files make the table.

Two table types. Copy-on-Write rewrites the base Parquet file of a file group on every change; reads are simple because there is nothing to merge. Merge-on-Read appends changes to log files beside the base file; a snapshot read merges them on the fly, a read-optimised read takes the base files only, and compaction periodically folds the logs into new base files. Merge-on-Read gives the shorter write latency, Copy-on-Write the faster read. Both support three query types: snapshot, incremental — latest state, or change data capture with before and after images — and, for Merge-on-Read, read-optimised.

Infographic

Receivables table: two ways to keep it current

Nightly full reload

• –Drop and rewrite the month's partition from the ERP
• –Hours of cluster time regardless of how much changed
• –Answers are as old as the last run
• –Downstream jobs recompute everything
• –A deletion means another full rewrite

Incremental upserts with Hudi

• →CDC events from the ERP log, applied by key
• →Cost proportional to the number of changes
• →Answers minutes behind the database
• →Downstream jobs read only what changed since their last instant
• →A deletion is one operation by key, final after cleaning

The path from the ERP

The usual chain is the database’s transaction log, read by Debezium into Apache Kafka, consumed by Hudi Streamer — the project’s own ingest tool, which supports Kafka, file systems, JDBC, Pulsar, Kinesis, object-store events and Debezium change records, runs in a continuous mode and reads schemas from a schema registry. The Streamer applies the changes as upserts to a Merge-on-Read table keyed by the ERP’s primary key.

From there the incremental query does the rest. A receivables job asks for every change since the instant it last processed and recomputes only the customers that moved. An agent that prepares credit decisions reads the snapshot and records the instant it read. Both are cheaper than the nightly reload and both can say what they saw.

Several writers can share such a table. Hudi 1.x adds non-blocking concurrency control for Merge-on-Read tables with a bucket index: writers take a lock only for timeline metadata, not for the data, and the project documents that this does not yet combine with clustering. The default is still a single writer, and optimistic concurrency control with a lock provider sits in between.

A distributor with credit limits

Consider a distributor whose ERP holds orders, invoices and credit limits, and whose sales desk needs the credit position within minutes of a payment being booked. The illustration is ours, not a client’s.

Before: the receivables table is rebuilt at night. Sales calls finance for anything urgent, and finance opens the ERP. A customer over the limit keeps ordering until the next morning.

With Hudi: Debezium emits invoice and payment changes; Hudi Streamer upserts them into a Merge-on-Read table keyed by invoice; a compaction schedule folds the log files every hour; an incremental job updates the open position per customer; a credit agent reads that position and proposes release or hold, which a person confirms in the ERP. When a customer exercises the right to erasure, the deletion is a write by key, and it becomes final when the cleaner removes the old file versions — a retention setting that someone has to own.

What to measure: the lag from a change in the ERP to its visibility in the table, the share of downstream compute that is incremental rather than full, the time to complete an erasure request, and the cost of the table services per month. The last one is the honest price of the minutes.

Where the engineering decisions sit

Coding agents handle a good part of this chain well. They generate the Streamer configuration from an event schema, draft the compaction and cleaning schedules, write the incremental job and the tests that compare an incremental result with a full recomputation on a sample. That comparison is the acceptance test we insist on before an incremental pipeline replaces a reload.

Three decisions should not be delegated to them. The record key, because a wrong key makes every later update a new row. The table type, because Merge-on-Read without an owned compaction schedule degrades quietly. And retention, because the cleaner is both the cost control and the moment an erasure becomes real. These go into the data specification with a name against each.

One more thing belongs there: who reads the table. The Hudi documentation lists Spark, Flink, Trino and Presto as engines. Other platforms generally reach a Hudi table through Apache XTable, which translates its metadata into Iceberg or Delta Lake — and XTable does not yet support Merge-on-Read tables. If a Snowflake or Fabric reader is in the picture, that constraint decides the table type before any benchmark does.

What a good outcome looks like

One table that the sales desk, the finance job and the credit agent all read, minutes behind the ERP, with a published lag figure. Downstream jobs that process changes rather than months. An erasure procedure with a date on it. And a line in the budget for compaction and cleaning that nobody is surprised by.

Our part

In AI data engineering we build the incremental path from the system of record: the CDC contracts, the Hudi tables with their keys, types and services, the incremental jobs with their acceptance tests, and the retention rules with named owners. We do not resell platforms. More about predictes is on the home page.