Columnar File Formats and Partitioning: Parquet, ORC and Avro

Analytical data is usually stored in columnar file formats such as Apache Parquet and Apache ORC, which store each column's values together, compress them well and let queries read only the columns and row ranges they need. Apache Avro is a row-based format used mostly for events and messages, where whole records are written and read at once. Choosing the format, and how files are partitioned and sized, often matters more for query cost than the engine you run.

Row vs columnar storage

Consider a table of orders with 40 columns. A report asks for revenue by country, which needs two columns.

Parquet vs ORC vs Avro

How columnar files skip work

• Column pruning: the reader decodes only requested columns.
• Row groups and statistics: each file is split into row groups, and each column chunk stores min, max and null counts. A query for order_date = '2026-10-01' can skip row groups whose min and max exclude that date. This is predicate pushdown.
• Encodings: dictionary encoding stores repeated strings once; run-length encoding compresses long runs of equal values. Sorting data by a common filter column makes statistics far more selective.
• Compression codecs: Snappy is fast and light; Zstandard compresses better at moderate CPU cost; choose by whether queries are CPU-bound or I/O-bound.

Partitioning

Partitioning splits a table into directories by a column value, such as event_date=2026-10-01/. A query filtering on that column reads only matching partitions.

• Partition by columns most queries filter on, usually a date, sometimes a region or tenant.
• Keep cardinality low. Partitioning by user_id creates millions of tiny directories and makes every query slower.
• Avoid partitions that are too small. Aim for partitions holding at least hundreds of megabytes; otherwise partition more coarsely (by month instead of day).
• Combine with sorting or clustering inside partitions for columns that are filtered often but have high cardinality.
• Modern table formats such as Iceberg support hidden partitioning, where queries filter on the original column and the format maps it to partitions automatically.

The small files problem

Streaming jobs and frequent small batches write many tiny files. Each file costs a metadata lookup, an open and a footer read, so a table of a million 100 KB files is far slower to query than the same data in a few thousand files of 128 MB to 1 GB. Fix it by buffering writes, reducing output partitions before writing, and running regular compaction jobs that rewrite small files into large ones.

A good default for analytics: Parquet files of roughly 128 MB to 1 GB, partitioned by date, sorted by the next most common filter, compressed with Snappy or Zstandard, and managed by a table format.