Parquet or Avro? The contract between systems and the cost of the answer

A distributor runs an ERP, a warehouse system and a web shop. Each night a job exports orders from the ERP as CSV, the warehouse imports them, and an analyst loads the same file into a reporting database. One Friday the shop adds a discount field. The export grows by one column, the warehouse import fails on Saturday, and the Monday margin report is built from Thursday’s data. Nobody changed anything wrong. The data simply had no contract and no layout suited to either job.

The two file formats most often proposed as the fix answer two different questions. Avro answers “will the next system read what this one wrote, including after a change”. Parquet answers “how little does a query have to read to return its result”. An AI-native enterprise system needs both answers, and most of the trouble comes from asking one format the other’s question. How the two file formats sit under table formats and operational databases is the subject of the series guide; this article is about the choice between them.

Two layouts, two jobs

Apache Avro writes records one after another, each complete, in a compact binary encoding that carries no field names in the data. The schema is a JSON document and travels with the data: in the header of an Avro container file, or as an identifier when single records move through Kafka and a schema registry. The current release is 1.12.2 (checked 2026-10-21). Writing one event costs one record; reading one event costs one record.

Apache Parquet writes all values of one column next to each other. A file is split into row groups, each row group into one chunk per column, each chunk into pages, and the footer holds the schema and statistics for every chunk. A query that asks for two columns of forty reads two columns of forty, and skips whole row groups whose minimum and maximum values exclude the filter. The format specification is at 2.14.0 (September 2026, checked 2026-10-21). Writing one event means rewriting a file; reading a million events for two columns is cheap.

	Apache Avro	Apache Parquet
Layout	row by row	column by column
Natural unit	one event, one record	one table, one scan
Typical write	continuous, single records	batches, whole files
Typical read	the whole record	a few columns of many rows
Schema	JSON, with the data or in a registry	in the file footer
Schema change	formal reader/writer resolution rules	handled by the table format and engine
Where it lives	Kafka topics, integration, event logs	lakehouse tables, warehouses, model training

Where the contract matters

Between systems, the risk is not speed. It is that a producer changes a message and a consumer that nobody told breaks on a weekend. Avro’s specification defines how a reader with one schema resolves data written with another: fields are matched by name, a field missing on the reader’s side is ignored, a field missing on the writer’s side takes the reader’s default or fails, and a fixed list of type promotions is allowed, such as int to long. A schema registry, such as Confluent’s, turns those rules into a gate: a producer cannot register a schema that would break the consumers under the compatibility policy the owner chose.

That is a contract in the legal sense of the word. The discount field from the opening scene is added with a default value, the registry accepts it as backward compatible, the warehouse consumer keeps reading without the field until its own release, and the margin report picks it up when the analytics side is ready. The change happens once, in a place where somebody is accountable for it.

Parquet has no equivalent role here. It can carry the same event, but a Parquet file written every few seconds by a stream produces thousands of small files, each with its own footer, and the layout that makes a scan cheap becomes the thing that makes it slow. Parquet is for data that has stopped moving.

Where the cost of the answer matters

Once events have landed, the questions change. Margin by channel and product group over a quarter. Return rate by supplier batch. Features for a demand model over two years of shipments. These questions touch millions of rows and a handful of columns, and in a cloud engine you pay for the bytes read and the time spent reading them.

This is Parquet’s job. Column pruning means the engine fetches only the requested columns. Statistics in the footer, the page index and Bloom filters let it skip row groups, pages and files that cannot contain the answer. Dictionary and run-length encoding followed by a codec such as ZSTD or Snappy shrink what remains; the layout decisions behind that are the subject of the Parquet profile. The same report run over Avro files reads every byte of every record, because a row layout gives the engine nothing to skip.

Infographic

Which format for which data

Many rows, few columnsOne record at a time

Avro in the streamEvents between systems, Kafka topics, CDC from the database. The schema registry is the contract.

Micro-batches into tablesStreams land in Parquet under a table format; compaction turns small files into scannable ones.

Avro as an archiveWorkable for replaying events in full, poor for any question that needs two columns of forty.

Parquet in the lakehouseReports, model features, agent reads. Column pruning and statistics decide the bill.

Data flowsData rests

The handover, and what the AI SDLC adds to it

In practice the architecture is a handover: Avro in the stream, Parquet at rest, with a table format such as Apache Iceberg or Delta Lake deciding which Parquet files make up a table at any moment. The decision that costs the most is the latency of that handover. Writing to the table every few seconds gives fresher data and a compaction bill; writing every few minutes gives larger files and a short delay that most reports never notice. Which decisions in the business genuinely change within a minute is a question for the owner of the process, not for the engineer.

Two things change when the pipelines are built with coding agents and treated as software with a lifecycle. The first is that the contract comes before the code: the Avro schema, its defaults and the compatibility policy are the specification, and the agent generates producers, consumers and compatibility tests from it rather than the other way round. The second is that the handover gets a test suite of its own: counts and sums of the same events read from the topic and from the Parquet table, a check that the latency stays under the agreed bound, and a check that no small-file backlog is building. Agents are good at writing and running those tests. Approving a schema change that drops a field, or moving the latency bound from minutes to seconds, stays with a person who can be asked about it later.

What a good outcome looks like

The schema of every event that crosses a system boundary is in a registry with a named owner and a compatibility policy. Adding a field is a routine release, not an incident. Reports and models read Parquet tables, and the bytes scanned per report are known and falling, not unknown and rising. There is exactly one path from the stream to the table, with a stated latency, and the number of small files is monitored. When somebody proposes JSON “to keep it simple”, the team can show what the lack of a schema cost the last time.

Our part

In AI data engineering we design that handover for one significant flow first: the event contracts in Avro, the landing tables in Parquet under an open table format, the tests that compare the two, and the agents that keep the pipelines in shape. We do not resell platforms. More about predictes is on the home page.