Apache Parquet: why the analytics bill depends on the file layout

A distributor’s finance team adds one dashboard: return rate by supplier batch, refreshed hourly. The query touches three columns of a shipments table with forty. The next invoice from the cloud engine is markedly higher, and the explanation offered is “more queries”. The real explanation is in the files: the table was written by a stream in small files, never compacted, never sorted, and every hourly refresh reads most of it.

Apache Parquet is the file format under nearly every lakehouse table — Delta Lake, Apache Iceberg, Apache Hudi and Apache Paimon all store rows in it by default. Its reputation for cheap analytics is deserved, but it is conditional. The bill depends on how the files are laid out, and the layout is a set of decisions somebody has to make and keep making. What Parquet is next to the operational database, and why the lakehouse is a second memory rather than a copy of the first, is covered in the series guide.

What is inside a Parquet file

The format specification is at 2.14.0 (September 2026, checked 2026-10-21). A file is a sequence of row groups. Each row group holds one column chunk per column, each chunk is a sequence of pages, and the footer — written last, so a file is produced in a single pass — holds the schema, the location of every chunk and its statistics. Three further structures matter for cost:

Statistics and the page index. Each column chunk records minimum, maximum and null counts; the page index records the same per page, with page offsets, so an engine can skip pages rather than only chunks.

Bloom filters. For high-cardinality columns such as an order identifier, a split-block Bloom filter per column chunk answers “this value is certainly not here” without reading the chunk.

Encodings and codecs. Values are first encoded — dictionary, run-length and bit-packing, delta encodings, byte-stream split — and the pages are then compressed with a codec: Snappy, GZIP, Brotli, ZSTD, LZ4_RAW. The choice is a trade between file size and the processor time spent on either side.

The logical type system keeps growing: version 2.11 added the VARIANT type for semi-structured data and GEOMETRY/GEOGRAPHY, 2.12 finalised shredding for variants, and 2.14 added a FILE type and a preview of the ALP floating-point encoding, according to the project blog. For an enterprise system this means JSON payloads from an ERP and location data from a fleet can live in the same table as the ledger, typed, rather than as opaque strings.

What the engine skips, and when it cannot

A cloud engine charges for bytes read and seconds spent. Parquet lowers both by giving the engine reasons to skip. Each reason depends on a property of the files that is not automatic.

Infographic

What a well-laid-out table lets the engine skip

1. 01Files, by partition and table statisticsThe table format's metadata excludes whole files; needs sensible partitions and files large enough to be worth listing.
2. 02Row groups, by min/max statisticsWorks only if the data inside a file is sorted or clustered on the filter column; random order defeats it.
3. 03Pages, by the page indexFiner skipping within a row group; written by modern writers, used by modern readers — both must be true.
4. 04Columns not asked forAlways available in a columnar layout; the largest and most reliable saving.
5. 05Repeated values, by dictionary encodingStatus codes, channels and categories compress to a few bytes per value; free, if the dictionary does not overflow.

Return to the hourly dashboard. The stream wrote a file every minute, so a month of shipments is some forty thousand files of a few hundred kilobytes. The engine lists all of them, opens each footer, and finds that every row group spans every supplier batch, because rows arrived in order of time, not of batch. Nothing can be skipped except the unused columns. Compacting the month into a few hundred files and sorting each by supplier batch changes which layers in the picture apply, and that is the difference on the invoice. The figures are illustrative; the mechanism is not.

Decisions, and who makes them

Layout is a set of decisions with owners, not a setting.

Partitioning — by what, and how coarse — is decided by how the table is queried and how much data each partition will hold. Too fine produces small files; too coarse produces full scans. This is a design decision made with the people who own the questions.

Sort or clustering keys within files decide whether statistics do anything. They follow the dominant filters, which change as the business changes.

File size and compaction schedule decide how many footers an engine opens. The table format provides the operation; somebody has to run it, pay for it, and notice when a stream has outrun it.

Codec and encoding decide the size-versus-processor trade; ZSTD usually wins on size, Snappy on speed, and the right answer depends on which side is paid for.

Retention decides how much history is kept in the hot table and what moves to cheaper storage. It is also where legal obligations arrive, because deleting a person’s records under GDPR means rewriting the files that hold them.

Agents in the data engineering loop

Most of these decisions are measured, not guessed, and measurement is where coding and operations agents earn their place. In the way we build this, an agent reads the engine’s query history and the table’s file statistics and reports which columns are filtered most, how many files a typical query opens, how skewed the file sizes are, and what the scanned bytes would be under a different sort key. It drafts the compaction job and the sort order, generates the before-and-after comparison on a sample, and opens a change request with the numbers attached.

What it does not do is approve the change. A new partitioning rewrites the table and changes every downstream job; a retention change may delete something an auditor will ask for; a codec change moves cost from storage to compute and shows up on a different team’s budget. These are reviewed by an engineer and, where they touch retention or deletion, by the owner of the data. The agent’s output is evidence; the decision stays recorded under a person’s name, which is also what the table format’s history will show.

Treated as part of an AI SDLC, the layout has tests like any other code: scanned bytes per reference query under a threshold, small-file count under a bound, page index and statistics present in every file written. A pipeline that silently starts writing unsorted files fails a test, not a quarterly budget review.

What a good outcome looks like

Every table in the lakehouse has a stated partitioning, sort order and file-size target, and a job that keeps them true. Scanned bytes per reference query are measured and published alongside the query latency. The compaction and retention schedules have owners and budgets. New columns from semi-structured sources arrive typed, not as strings. When a new dashboard raises the bill, the team can say within a day whether the cause is the question or the layout.

Our part

In AI data engineering we take one significant table through this: measured layout decisions, the compaction and sorting jobs, the tests that keep scanned bytes down, and the agents that watch the files so engineers can decide rather than search. We do not resell platforms. More about predictes is on the home page.