Apache Avro: the schema is the contract your systems sign

An agent that prepares credit decisions for a distributor reads three streams: orders from the web shop, shipments from the warehouse, payments from the ERP. On a Tuesday the ERP team renames a field from customer_no to customer_id and retypes it. The producer keeps running. The agent keeps running too, and for four hours it assigns every payment to an unknown customer, which it reports as overdue. No log line says anything failed.

That incident is the reason Apache Avro exists in an enterprise integration. Avro is a row format, and it is a reasonable way to serialise an event compactly, but its job in a system of record is a different one: it is the place where the contract between two systems is written down, checked by a machine, and changed only by somebody who is accountable for it. Where that contract sits between the operational database and the lakehouse is laid out in the series guide.

What Avro consists of

Four parts of the Avro specification matter for an architect; the current release is 1.12.2 (checked 2026-10-21).

The schema. A JSON document with primitive types (null, boolean, int, long, float, double, bytes, string) and complex ones (record, enum, array, map, union, fixed). Fields can carry a default value and documentation. The schema is the thing a team argues about, and that is as it should be.

The binary encoding. A record is written as its field values in schema order, with no field names and no type tags inside the data. That is why Avro records are small and why the reader must have the schema the writer used.

The container file. For data at rest, Avro’s own file starts with the bytes Obj and a version byte, stores the schema in its metadata, and writes records in blocks separated by a 16-byte sync marker, so a file can be split and read in parallel. null and deflate codecs are required by the specification; snappy, zstandard, bzip2 and xz are optional.

Single-object encoding. For one record moving through a message broker, the specification defines a two-byte marker, an eight-byte fingerprint of the schema and then the payload, so the reader can look the schema up. Confluent’s registry uses its own framing for the same purpose: a magic byte, a four-byte schema identifier, then the Avro bytes (documentation).

How a reader survives a writer’s change

The part of Avro that earns its place is schema resolution: the rules for a reader holding one schema to read data written with another. Fields are matched by name. A field the writer wrote and the reader does not know is skipped. A field the reader expects and the writer did not write takes the reader’s default value; without a default, reading fails. A short list of promotions is allowed — int to long, long to float or double, float to double, string and bytes in both directions. An unknown enum symbol falls back to the reader’s default if one is declared.

Read against the Tuesday incident, the rules say exactly what should have happened. Renaming customer_no to customer_id is, to a reader, one field disappearing and another appearing. Without an alias or a default on the reader’s side, that is a failure, and a failure is the correct result: loud, at the first message, in the producer’s release pipeline rather than in a credit decision four hours later. Avro provides aliases for renames and defaults for additions; a team that uses them turns most changes into non-events and the remaining ones into visible decisions.

A schema registry makes the rules enforceable. Each subject has a compatibility policy — backward, forward, full, or their transitive variants — and a producer cannot register a schema that violates it. The policy belongs to the owner of the event, who is usually the team that owns the system of record it describes, not the team that happens to be writing the consumer this quarter.

Where Avro stops

Avro is poor at the questions the lakehouse exists for. A report over a year of shipments that needs three columns of thirty reads every byte of every record, because a row layout offers nothing to skip. Model features, margin by channel and return rates belong in Parquet under a table format, and Avro’s role is to carry each event there intact. The usual pattern is a stream processor — Flink or Spark — that reads the Avro topic and writes Parquet tables in Delta Lake, Apache Iceberg or Apache Paimon at an agreed latency. Why the two formats divide the work this way is the subject of Parquet or Avro?.

Avro also appears in a place few people expect: Apache Iceberg writes its manifest files — the lists of data files that make up a table snapshot — in Avro, precisely because they are read record by record and must survive schema changes of their own.

The schema change in an AI SDLC

Treating integration as software with a lifecycle changes who does what when a field changes. In the engagements we run, the sequence is fixed, and coding agents do most of the typing in it.

Infographic

The path of one schema change

1. 01ProposalAn agent drafts the new schema from the change request, with defaults and aliases, and lists every consumer the registry knows.
2. 02Compatibility checkThe registry evaluates the schema against the subject's policy; the agent generates and runs resolution tests for each consumer's reader schema.
3. 03Owner approvalThe owner of the event reads the diff and the test results and signs the change. This step is a person, not a pipeline stage.
4. 04ProducersThe producing system is released with the new schema; old consumers keep reading under the resolution rules.
5. 05ConsumersEach consumer adopts the new field on its own release schedule; the registry records which version each reads.

The agents are useful because the work is mechanical and tedious for people: enumerating consumers, writing a reader-schema test per consumer, generating serialisation code in three languages, keeping the documentation field of each attribute current. They are dangerous in one place: an agent asked to “make the pipeline pass” will happily loosen a compatibility policy or add a permissive union. The registry policy and the approval step are therefore configuration that agents can read but not change.

Consider a distributor with the three streams from the opening. After the change, the ERP’s payment_posted event is version 7 in the registry; the credit agent reads version 5 with defaults for the two fields it does not use yet; the lakehouse landing job reads version 7. A new field for a payment channel is added on a Thursday and reaches the margin report the following week, with no weekend involved. The figures are not the point of the example; the absence of an incident is.

What a good outcome looks like

Every event that crosses a system boundary has one schema in one registry, with an owner and a compatibility policy. Incompatible changes fail in the producer’s pipeline, not in a consumer at night. Adding a field is a release, not a project. The lakehouse receives every event intact, with its schema version recorded, so a question about what an agent saw on a given day can be answered from the data rather than from memory.

Our part

In AI data engineering we put the contracts in place for the first significant flow: the Avro schemas with their defaults and aliases, the registry and its policies, the resolution tests generated per consumer, and the landing into open table formats. We do not resell platforms. More about predictes is on the home page.