Parquet vs ORC vs Lance vs Vortex: Columnar File Formats Compared

For almost a decade the Parquet vs ORC question was the whole conversation about columnar file formats, and the answer was usually “Parquet, because everything reads it.” That answer is still mostly right for batch analytics. But the workloads landing on object storage in 2026 are different: wide tables with thousands of columns, embedding vectors, images and tokenized text sitting next to scalars, and training jobs that need random rows at NVMe speed rather than sequential scans at warehouse speed. Two newer formats, Lance and Vortex, were built specifically because Parquet’s layout makes those workloads expensive.

This post explains how the four formats actually lay bytes out, why their encodings behave so differently on the same data, and where each one wins or loses. You will leave with a mental model of row groups, pages, stripes and structural encodings, a worked sizing example for dictionary and bit-packing, a decision matrix, and a tuning checklist that avoids the small-files and oversized-row-group traps. Vendor performance claims are separated from independently published results throughout.

What this covers: the common columnar model, Parquet and ORC internals, predicate pushdown through statistics, Lance’s adaptive structural encodings, Vortex’s cascaded encodings, a comparison matrix, failure modes, and a practical selection guide.

Context and Background

Columnar storage rests on one observation: analytical queries touch a few columns of many rows, so storing each column contiguously lets you read only what you need and compress each column with a scheme matched to its values. Apache Parquet and Apache ORC both implement this, and both descend from the same ideas: Google’s Dremel paper for nested data in Parquet’s case, and the Hive RCFile lineage for ORC. Parquet became the interchange default, helped by Spark, Arrow, Trino, DuckDB, Snowflake, BigQuery and the Iceberg and Delta table formats. ORC remained strong in Hive-centric stacks, where its built-in indexes, bloom filters and ACID support in Hive were designed together.

Two forces are now pulling on that settled picture. First, hardware: NVMe drives deliver hundreds of thousands of random 4 KiB reads per second, and object stores deliver high aggregate bandwidth, so the bottleneck has moved from disk to CPU decoding and to the number of round trips. Second, data shape: ML pipelines want to fetch individual rows by index, shuffle, and project large binary columns, which row-group-oriented formats handle poorly.

The ecosystem has begun to acknowledge this. The Apache Parquet format specification now lists an ALP encoding for floating-point values, though its own documentation marks it as being in preview. A September 2026 lakehouse state-of-the-union from Dremio reports that the Parquet format has reached 2.13.0, that a policy adopting major versions as the vehicle for forward-incompatible changes was voted in during August 2026, and that ALP, FSST and a FlatBuffer-based footer are in the pipeline. Treat those as reported by a vendor blog rather than verified against release tags. Iceberg 1.11.0 (reported release date May 19, 2026) added a File Format API so that engines can plug in formats beyond Parquet, ORC and Avro without rewriting every read path. That matters directly for Lance and Vortex adoption inside lakehouse tables; see our walk-through of the Apache Iceberg v3 spec and features for how the table layer sits above these file formats.

On the newcomer side, Lance began as the storage layer of the LanceDB vector database and has a peer-reviewed-style description in the arXiv paper “Lance: Efficient Random Access in Columnar Storage through Adaptive Structural Encodings” (arXiv 2504.15247). Vortex was created by SpiralDB and contributed to the Linux Foundation’s LF AI & Data Foundation as an incubation-stage project, announced on August 6, 2025, with Microsoft, Snowflake and Palantir named as supporters. As of this writing, the Python package pylance shows version 13.0.0 (October 7, 2026) on PyPI and the vortex Rust crate shows 0.90.0 (October 23, 2026) on docs.rs. Both version numbers are quoted from those registries on the day of writing; check them again before you pin anything.

One framing point before going deeper. “Which format is fastest” is the wrong question, because the four formats make different bets on the same four-way trade-off: scan throughput, point-lookup latency, compression ratio, and ecosystem reach. Every section below returns to that trade-off.

The Columnar Model: How Each Format Actually Lays Out Bytes

A file format is a contract between a writer and many readers. The useful way to compare formats is to ask three questions: what is the unit of skipping, what is the unit of decoding, and where does the metadata live. Parquet and ORC answer these similarly but not identically. Lance and Vortex answer them differently on purpose.

Figure 1: Parquet organizes data into row groups, column chunks and pages with a footer at the end; ORC organizes it into stripes that each carry their own index, data and footer.

Figure 1 shows the two classic layouts side by side. Both are written in a single pass, with the file-level metadata at the tail, so a reader’s first action is to fetch the end of the file. Both choose a horizontal partition (row group or stripe) first, then store columns contiguously within it. That horizontal partition is the unit at which statistics are collected and at which a reader can skip data without touching it.

Parquet: row groups, column chunks and pages

The Apache Parquet documentation describes the layout this way: the file starts and ends with the 4-byte magic number PAR1; the body is M row groups; each row group holds one column chunk per column, written in column order; and the footer metadata, placed after the data to allow single-pass writing, records the start location of every column chunk. The footer ends with a 4-byte little-endian length and the closing magic. A reader seeks to the end, reads the length, reads the footer, and then knows exactly which byte ranges to fetch.

Inside a column chunk, data is split into pages. A chunk may begin with a dictionary page, followed by data pages. Each data page holds repetition levels, definition levels and encoded values. Repetition and definition levels are how Parquet represents nesting and nulls in the Dremel style: definition levels say how many optional ancestors are present, and repetition levels say at which list depth a new element begins. These levels are themselves encoded with an RLE/bit-packing hybrid, which is why a mostly non-null flat column pays almost nothing for them.

The practical defaults come from the writer library, not the specification. In parquet-java the commonly cited defaults are a 128 MiB row group target, a 1 MiB page size and a 1 MiB dictionary page limit; other writers (parquet-rs, Arrow C++, DuckDB, Spark’s own configuration) differ. Treat those figures as typical-but-writer-dependent and verify the setting in the library you actually run. The spec itself gives guidance rather than mandates, which is one reason files from different engines behave differently.

The ORC v1 specification describes a file body made of stripes. Each stripe has index streams first, then data streams, then a stripe footer, and contains only whole rows. The specification says stripes are “typically ~200MB”; the ORC Java writer’s configurable default is smaller, and I have not verified its current value here, so check orc.stripe.size in your version. Within a stripe, the row index stride defaults to 10,000 rows, and each such slice is an ORC “row group” with its own entry in the row index: per-group min, max, sum, null counts and stream positions.

ORC’s column encodings are named by kind: DIRECT and DICTIONARY use the older RLE v1, while DIRECT_V2 and DICTIONARY_V2 use RLE v2. RLE v2 is the interesting part: it switches among four sub-encodings per run of integers. Short Repeat handles short runs of an identical value, Direct handles arbitrary values at a fixed bit width, Patched Base handles values with a few outliers by storing a base plus patch list, and Delta handles monotonic sequences. That adaptivity is built into the format rather than left to the writer, which is a notable design difference from Parquet. The supported compression codecs are NONE, ZLIB, SNAPPY, LZO, LZ4 and ZSTD, applied to chunks of 256 KiB by default, each with a 3-byte header; chunks that do not compress are stored raw.

Vocabulary map between the two

The terms are confusing because they overlap. A Parquet row group (hundreds of MiB, the unit of skip-by-statistics and of parallel split) corresponds loosely to an ORC stripe. A Parquet page (about a MiB, the unit of decompression and of the optional page index) corresponds loosely to an ORC row index group of 10,000 rows, though the ORC group is a logical row range across streams while a Parquet page is a physical blob inside one column chunk. Neither mapping is exact, and that inexactness matters when you tune: shrinking Parquet row groups changes parallelism and footer size, while shrinking ORC’s index stride changes index granularity and metadata overhead.

Where Lance and Vortex depart from this model

Lance’s file layout is, in its own documentation, an encoding-agnostic container. Its file documentation states that a default disk page size of 8 MB is recommended, that buffers are aligned (64 bytes in practice, and 4096 bytes for direct I/O), and that the footer carries three 64-bit offsets, buffer and column counts, major and minor version fields and the magic string LANC, for 40 fixed bytes. A reader needs one sector from the end of the file for the footer, which the docs describe as one or two I/Os, or one when the metadata size is stored in the table manifest. Critically, the Lance 2.x file format removed row groups entirely (documented as the 2.0 rework), so there is no fixed horizontal partition at all; the unit of data is the page of a single column, and the structure inside a page is chosen per column by the writer.

Vortex describes itself as an extensible framework for columnar compression and file format, separating the logical schema from the physical layout, with pluggable encodings and a “cascading compression” mechanism in which the output of one encoding can be fed to another. Its README states the file format is considered stable from release 0.36.0 onward: files written by any version from 0.36.0 can be read by later releases, though library APIs may change between versions. That is a useful compatibility statement, and also a reminder of how young the project is next to a format with ten years of production history.

Encodings, Compression and Predicate Pushdown: The Mechanisms That Decide Performance

Encoding is the step that turns typed values into compact bytes while keeping them cheap to decode; compression is the generic byte-level step (Snappy, LZ4, ZSTD) layered on top. Most of the performance difference between formats is not the container at all. It is which lightweight encodings the format provides, and who chooses among them.

Parquet encodings in practice

The Parquet encodings table lists these, with enum values from the spec:

Encoding	Enum	Applies to	What it does
PLAIN	0	all physical types	values stored back to back, little-endian
RLE_DICTIONARY	8	all physical types	per-chunk dictionary plus RLE/bit-packed indices
RLE	3	booleans, dictionary indices	run-length plus bit-packing hybrid
DELTA_BINARY_PACKED	5	INT32, INT64	block deltas with per-miniblock bit width
DELTA_LENGTH_BYTE_ARRAY	6	BYTE_ARRAY	delta-coded lengths, then concatenated bytes
DELTA_BYTE_ARRAY	7	BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY	front coding of shared prefixes
BYTE_STREAM_SPLIT	9	INT32, INT64, FLOAT, DOUBLE, FLBA	splits value bytes into per-position streams
ALP	10	FLOAT, DOUBLE	decimal-like floats to integers with exceptions (preview)

PLAIN_DICTIONARY and BIT_PACKED are deprecated. Two Parquet-specific details trip people up. First, the dictionary is per column chunk, so a column with high cardinality overflows the dictionary page limit partway through a chunk, and a conforming writer falls back to PLAIN for the remaining pages. The result is a column chunk whose early pages are dictionary-coded and whose later pages are not, which is perfectly valid and quietly inflates file size. Second, which encodings you get depends on the writer version setting: many engines default to the older “v1” data page and encoding set for compatibility, and the delta encodings only appear when the writer is configured to use the newer set. The set of parquet encodings your files really contain is therefore a property of how they were written, not of the Parquet spec.

BYTE_STREAM_SPLIT deserves attention for scientific and sensor data. A float64 stored plain interleaves exponent bytes, which are highly repetitive, with mantissa bytes that look random. Splitting into eight byte streams groups the repetitive bytes together so ZSTD or LZ4 can find them. It adds no compression by itself; it is a transform that makes the downstream codec effective. This is directly relevant to the telemetry workloads this site covers, such as time-series from IoT fleets.

A worked example: dictionary plus bit-packing

Consider a column of a million rows holding a device status string drawn from 1,000 distinct values averaging 12 bytes each. The numbers below are illustrative arithmetic, not a benchmark.

Plain encoding stores a 4-byte length prefix and 12 bytes of data per value in Parquet’s BYTE_ARRAY layout: 16 bytes times one million is about 16 MB per column. Dictionary encoding stores the 1,000 distinct values once, about 16 KB, and one index per row. A dictionary of 1,000 entries needs ceil(log2(1000)) = 10 bits per index. Bit-packed that is 10 bits times one million, which is 1.25 MB, plus a few percent hybrid overhead. The dictionary-coded column is about 1.27 MB versus 16 MB, a ratio near 12.6 to 1, before any Snappy or ZSTD pass. If the statuses arrive in long runs, the RLE half of the hybrid collapses each run to a (count, value) pair and the index stream shrinks further. If the column instead had 5 million distinct values across 10 million rows, the dictionary itself would be large, the dictionary page limit would trigger fallback, and the benefit would largely vanish. Cardinality is the variable that decides.

The same arithmetic explains why shrinking row groups costs compression: the dictionary is rebuilt per chunk, so a 1,000-value dictionary written into every 5 MB chunk is paid many times, and the cross-chunk repetition is never exploited. That is the first of several tuning trade-offs, covered below.

Lightweight versus general-purpose compression

Snappy and LZ4 are fast and give modest ratios; ZSTD at low levels gives better ratios at somewhat higher CPU; the heavier gzip/zlib family is usually dominated by ZSTD on modern hardware. The key insight from the research behind Vortex (the BtrBlocks and FastLanes papers cited in its README acknowledgments) is that stacking cheap, vectorizable encodings such as dictionary, frame-of-reference, bit-packing and delta can approach the ratio of general-purpose codecs while decoding at memory-bandwidth speed, and that the decoder can sometimes operate on the compressed form. The cost is selection complexity: someone has to pick the encoding cascade per column chunk, and that logic is the bulk of a Vortex-style writer. Parquet leaves the choice to the writer library; ORC bakes part of it into RLE v2.

Predicate pushdown: skipping before decoding

Figure 2: A pushdown-capable reader reads the footer, prunes row groups with min/max statistics, prunes pages with the page index, probes bloom filters for equality predicates, and only then range-reads the surviving column chunks.

Figure 2 shows the read path. The reader first fetches the footer, then tests each predicate against per-row-group (Parquet) or per-stripe and per-index-group (ORC) minimum and maximum values. A predicate such as ts BETWEEN a AND b on a column whose rows were written in time order eliminates most row groups from metadata alone, and the engine never issues a read for them.

ORC records richer typed statistics: the specification lists count and optional hasNull for every column; min, max and sum for integers and doubles; min, max and total length for strings; true and false counts for booleans; and per-row-group copies of these in the row index plus per-stripe copies in file metadata. It also defines bloom filters, added in Hive 1.2.0, as one stream per column listed in orc.bloom.filter.columns, one entry per row group, hashing numeric values with a Thomas Wang 64-bit hash and strings with 64-bit Murmur3. Parquet has equivalent pieces: per-chunk and per-page statistics, a column index and offset index (the “page index”) for page-level pruning, and split-block bloom filters for equality predicates. Both formats can therefore do hierarchical pruning; the differences are in what the writer enables by default and what the reader actually consults. Many deployments quietly lose pushdown because the writer disabled statistics for long strings, the reader ignores the page index, or the data is not clustered on the filter column.

The honest worked-numbers version of pushdown follows from clustering. If a one-terabyte table has 8,000 row groups of 128 MiB and a query filters a column that is sorted or tightly clustered, min/max may prune 99 percent of them, leaving about 80 row groups to read. If the same filter column is uniformly random, every row group’s min and max span nearly the whole domain and pruning removes close to nothing. The format cannot fix this; sorting or Z-ordering at write time (or table-level clustering in Iceberg or Delta) does. Statistics are only as selective as the physical order of the data.

Lance: Adaptive Structural Encodings for Random Access

Parquet’s design optimizes for scanning columns of many rows. To read row 1,234,567 of a column, a reader must locate the right row group, then the right page (using the offset index if one was written), then decompress the whole page and decode up to the target row. For small scalar columns that amplification is tolerable; for a 3 KB embedding or a 200 KB image it is painful. Lance was designed around this gap.

Direct answer: Lance is a columnar format whose writer picks one of two structural encodings per column depending on value width. Narrow values use mini-block chunks of a few KiB that compress well and scan fast. Wide values use full-zip layout with a repetition index, so any row can be fetched in at most two I/Os. Both aim to keep scan speed close to Parquet’s.

Figure 3: The Lance writer routes each column to a mini-block or full-zip structural encoding depending on average value width, trading chunk-level read amplification against index size.

Mini-block versus full-zip

The Lance paper (arXiv 2504.15247) explains that encoding a column’s structure, meaning its repetition and validity information, is central to exploiting the disk. It describes two structural encodings. The mini-block encoding splits an array into small chunks, each targeting roughly one to two disk sectors (4 to 8 KiB) of compressed data, with at most 4,096 values per chunk. Each chunk holds its repetition levels, definition levels and data buffers untransposed, so encoding and decoding stay vectorized and the compression can be opaque, meaning any scheme that compresses a chunk as a unit is allowed. The cost is that fetching one value requires decoding its whole chunk. The in-memory search cache costs 24 bytes per chunk, or 41 with a repetition index.

The full-zip encoding is used for values of 128 bytes or more, such as embeddings, tensors, images and long text. It zips a column’s repetition levels, definition levels and data into a single row-major buffer, with each value preceded by a control word of one to four bytes. A repetition index then allows random access in two I/Os regardless of nesting depth, and full-zip needs no search cache. Because values are compressed before zipping, compression must be transparent to the layout, which limits the choices, and nulls in fixed-width types require filler bytes.

The stated goals are at most one I/O for random access to a fixed-width column and at most two for a variable-width column, while keeping full-scan performance comparable to Parquet. The paper also describes struct packing, storing a struct as one column to speed multi-field random access at the cost of single-field projection.

What the paper reports, and what it does not

Some of the paper’s numbers are worth quoting because they cut against the usual narrative. The authors report that Parquet with default parquet-rs settings reached about 5,500 rows per second on small scalar random access, while optimized settings reached about 350,000 rows per second, and that correctly configured Parquet can achieve over 60 times better random access than its defaults. In other words, much of the “Parquet is slow at random access” folklore is really “default Parquet settings are slow at random access.” The disk baseline in their experiment was about 850 thousand random 4 KiB reads per second and about 3,400 MiB/s of throughput. They found that dictionary encoding reached only about 2 percent of ideal random-access performance while compression-oriented configurations reached about 88 percent, and that Parquet full scans reached roughly half of the disk’s peak bandwidth for many encodings.

Relative to Parquet, the paper reports Lance full-scan results (normalized, read from a figure and so approximate) of about 1.8 to 2.0 times for names, embeddings and images, about 1.3 to 1.5 times for reviews, prompts and code, slightly above parity for websites and slightly below for dates. Random access is described as generally tying or beating Parquet, with no exact figure given for that comparison. These are the authors’ own results on their hardware and datasets; they are a good illustration of the mechanism but not a substitute for measuring your data. The same paper estimates Parquet’s page offset index at about 20 bytes per page, or on the order of 20 GiB per billion rows for large columns, versus a maximum Lance search cache without repetition index of roughly 1.28 GiB per billion rows. That index-memory difference is what makes a billion-row random-access service practical.

Versions and the table layer

Lance file format versions matter operationally. Per the Lance versioning documentation, 2.0 reworked the format and removed row groups, 2.1 improved integer and string compression and sped up random access on nested fields, and 2.2 expanded nested type and encoding support including maps. A 2.3 version exists as unstable, with sparse structural pages and experimental encodings, and “stable” and “next” are aliases that move with your installed library. The documentation advises pinning an explicit version during rollouts, and notes at least one forward-compatibility break (all-null FixedSizeList inner values written by Lance 11.1.0 or later cannot be read by older versions). Pin versions. Lance also couples the file format to a dataset format with manifests, versioning and indexes; if your goal is a vector retrieval store, compare it against the options in our pgvector vs Qdrant vs LanceDB comparison.

Vortex: Cascaded Encodings and Compute on Compressed Data

Vortex attacks a different bottleneck: CPU time spent decoding. Its README describes it as an extensible framework for columnar compression and the fastest open-source columnar file format, aimed at object storage, with a pluggable encoding system and cascading compression. Its design acknowledgments cite BtrBlocks, FastLanes, FSST and ALP, which are academic encodings: BtrBlocks (a cascade of lightweight encodings chosen by sampling), FastLanes (a data layout enabling SIMD-friendly decoding of bit-packed and delta data), FSST (fast static symbol table compression for strings) and ALP (adaptive lossless floating-point compression). The README presents these as inspirations rather than an inventory of what ships, so I will not claim that every one is in the current release.

How a cascade works

A cascade applies one encoding, then treats its outputs as columns to encode again. Take a column of timestamps in milliseconds that mostly tick forward by a roughly constant amount. Delta encoding turns it into small, repetitive differences. Frame-of-reference subtracts a base so the values fit in few bits. Bit-packing stores them at the minimal width. A dictionary on top handles repeated deltas. Each stage is cheap and vectorizable, and the writer samples the data to decide the cascade depth. Because encodings are objects in the file with their own metadata, a reader can evaluate some operations without fully expanding the data: for example, comparing a dictionary-encoded column to a constant can be done by comparing against the dictionary, then mapping indices.

A third-party blog post (from urandom.io, May 2026, an opinion piece and not a measurement) reports that the DuckDB team describes Vortex running filters and expressions inside storage segments without fully decompressing first. I could not independently verify that statement, so treat it as reported. The mechanism is plausible and is the core reason to be interested: if the encoded form supports predicates directly, the CPU never touches most of the bytes.

The vendor performance claim

The Linux Foundation announcement and the project README both state a SpiralDB claim of “100x faster random access reads, 10-20x faster scans, and 5x faster writes compared to Apache Parquet,” with similar compression ratios. The announcement also reports that Microsoft demonstrated 30 percent runtime reductions on traditional Spark workloads using Vortex in Apache Iceberg. Neither source gives a methodology, hardware or dataset on the page I fetched. The project points to bench.vortex.dev for benchmarks. Until you reproduce comparable numbers on your data, treat the multipliers as the upper bound of a favorable case. Remember also the Lance paper’s lesson: a baseline of default-configured Parquet can understate what a tuned Parquet delivers by more than an order of magnitude on random access, so a “100x over Parquet” headline depends heavily on which Parquet.

Integration status

According to the Linux Foundation announcement, Vortex has native integrations with Apache Arrow, DataFusion, DuckDB and Spark, with Iceberg integration described there as coming soon at that time (August 2025); the Iceberg File Format API shipped in 1.11.0, per the Dremio post, which is the hook such an integration would use. It also claims to be the first storage format designed for direct GPU decompression. Whether a given engine in your stack can read and write Vortex today is something to check against that engine’s current documentation, not this post.

Trade-offs, Gotchas, and What Goes Wrong

Every format fails in characteristic ways. Knowing them is more useful than any benchmark.

Figure 4: Start from the workload and the table format; ecosystem constraints usually decide before benchmarks do.

Small files and metadata overhead

The most common production failure is not the format but the file count. Streaming writers that flush every minute produce thousands of 5 MB Parquet files, each with its own footer; planning cost grows with file count, object-store request latency dominates, and each file’s row group is too small for statistics or dictionaries to help. A rough guide, which is a heuristic and not a standard, is to compact toward files in the hundreds of MiB to low GiB range with row groups in the tens to low hundreds of MiB. Table formats automate compaction; plain directories of files do not. The same applies to ORC, where tiny stripes waste the index, and to Lance, whose manifest-based versions accumulate fragments until compacted.

Oversized row groups and memory

The opposite error is a 1 GiB row group on a wide table. A writer buffers an entire row group in memory before flushing, so 2,000 columns times many megabytes of per-column buffers can exhaust an executor’s heap. Readers that cannot split a row group also lose parallelism. Wide tables are where Parquet’s footer starts to hurt too: the Thrift-encoded footer must be fully parsed even to read two columns, so a table with tens of thousands of columns can spend more time parsing metadata than reading data. The FlatBuffer-based footer in the Parquet pipeline (reported by Dremio, not yet verified as released) targets exactly this.

Statistics that do not prune

Pushdown fails silently. Typical causes: data not clustered on the filter column; statistics truncated or disabled for long strings; timestamps stored in a representation whose ordering differs from the predicate’s; a reader that does not consult the page index; and writers that emit min/max for dictionary-fallback chunks incorrectly. Always verify with an engine’s scan metrics (rows skipped, bytes read) rather than assuming.

Encoding fallbacks and writer drift

Dictionary overflow, mixed encodings within a chunk, and differing writer defaults mean two “Parquet” files with the same schema can differ twofold in size. Spark, DuckDB, Arrow and Trino writers each choose differently. Compaction through a single writer normalizes this.

Nested data and schema evolution

Parquet’s Dremel levels make deeply nested, sparsely populated data costly to assemble. ORC keeps explicit per-type streams, which can be simpler for some nested shapes but couples evolution to column IDs. Lance 2.1 specifically targeted nested-field random access. Variant with shredding, described by Dremio as the production answer for semi-structured columns in Parquet and Iceberg v3, is the current way to avoid parsing JSON strings at query time.

Young-format risks

Lance and Vortex move quickly: version numbers of pylance 13.0.0 and vortex 0.90.0 in October 2026 indicate rapid iteration. Vortex’s own README says APIs may change between versions while the file format is stable from 0.36.0. Lance’s docs flag unstable file versions. Operational risks include thin tooling for inspection and repair, fewer engines with predicate pushdown into the format, and bus-factor concerns on projects governed by one main vendor. None of these are reasons to avoid them; they are reasons to pilot with a fallback path.

The benchmarking trap

Every format comparison hides choices: compression codec and level, row group size, page size, statistics settings, data clustering, cold versus warm cache, local NVMe versus object storage. A fair comparison fixes the dataset and queries, tunes each format to its own best practice, and reports bytes read and CPU time next to wall time. If a comparison does not say what Parquet settings it used, the headline multiplier carries little information.

Comparison Matrix and Choosing a Format

The matrix summarizes qualitative strengths based on the mechanisms above. It is a decision aid, not measured data.

Dimension	Parquet	ORC	Lance	Vortex
Primary unit	row group, column chunk, page	stripe, index group	column page, no row groups (2.x)	segments with cascaded encodings
Ecosystem reach	widest: Spark, Trino, DuckDB, Arrow, cloud warehouses	strong in Hive, Spark, Trino	LanceDB, Spark, DuckDB, Arrow, ML tooling	Arrow, DataFusion, DuckDB, Spark (per LF announcement)
Sequential scan	strong	strong	competitive per authors’ paper	vendor claims 10-20x over Parquet, unverified
Point lookup	weak by default, tunable	weak by default	designed for 1-2 I/Os	vendor claims large gains, unverified
Wide or vector columns	footer cost, read amplification	similar	full-zip layout built for this	designed for wide schemas, per README
Built-in indexes and bloom	stats, page index, bloom filters	stats, row index, bloom filters	separate index layer in dataset	evolving
Maturity	highest, versioned spec	high	moderate, versioned format with unstable tier	early, stable file format since 0.36.0
Lakehouse table support	Iceberg, Delta, Hudi native	Iceberg, Hive ACID	own dataset format; limited elsewhere	emerging via Iceberg File Format API

When to choose which

Choose Parquet when interoperability is the requirement: multiple engines, external partners, BI tools, or a table format that assumes it. Tune it, do not abandon it. The Lance paper’s 60x gap between default and tuned random access should make you suspicious of any claim that Parquet “can’t” do something before you have tried smaller pages, page indexes and optimized settings.

Choose ORC when your platform is Hive-centric, you rely on Hive ACID tables, or your engine’s ORC reader is demonstrably better tuned than its Parquet reader. In a Parquet vs ORC bake-off on a modern Spark or Trino stack, results on comparable settings are typically close, and the larger determinant is the reader implementation in your engine. Verify rather than assume either direction.

Choose Lance when you serve random rows or vectors, store large binary payloads next to scalars, or feed training loops that sample and shuffle. Its strengths are access patterns Parquet’s layout resists, plus a built-in versioned dataset with vector indexes. If your workload is large sequential SQL, Lance’s advantage shrinks.

Choose Vortex when you are CPU-bound on scans over fast storage, work with GPUs, or want to test compute on compressed data, and your engines already read it. Treat it as a pilot: convert a representative table, measure bytes read, CPU seconds and cost, and keep Parquet as the system of record until the ecosystem and your own measurements justify more. Analytical engine choice interacts with this; see our ClickHouse vs Doris vs StarRocks ADR for how engines with native storage differ from those that query open files.

Practical Recommendations

Start with the table format and engine, because they constrain the file format more than preference does. If you run Iceberg across several engines, Parquet remains the safe default, and the File Format API is the path by which others become usable. If you store ML datasets or embeddings, evaluate Lance on your access pattern before you build a custom cache layer.

Second, treat tuning as part of the decision. Most disappointing Parquet results come from three settings: row group size too small or too large, statistics or page index disabled, and data not clustered on the common filter column. Fix those before moving formats. For floating-point telemetry, test BYTE_STREAM_SPLIT with ZSTD; for low-cardinality strings, confirm that dictionary encoding is not falling back; for timestamps and counters, check for delta encodings by enabling the newer writer version if your readers support it.

Third, pilot newcomers with discipline. Pin library and file format versions, keep a Parquet copy, define success as a metric (bytes read, p99 lookup latency, CPU seconds per terabyte scanned), and run the same queries on identically clustered data.

Compact to large files and keep row groups within your executor memory budget.
Sort or cluster on the dominant filter column, then confirm pruning in scan metrics.
Inspect real encodings with parquet-tools, pyarrow metadata or orc-tool, not assumptions.
Pin Lance and Vortex versions; avoid “next” and unstable tiers in production.
Benchmark on your data with tuned baselines, and report bytes read alongside time.
Keep an exit path: convert back to Parquet should the pilot stall.

Frequently Asked Questions

What is the difference between Parquet and ORC?

Both are columnar, compressed formats with footers and statistics. Parquet partitions into row groups of column chunks and pages, uses Dremel-style repetition and definition levels for nesting, and has the broadest engine support. ORC partitions into stripes with built-in index streams, bloom filters and an adaptive RLE v2 integer encoding, and is strongest in Hive environments. On modern engines, performance is usually close and depends more on the reader implementation than the format.

Is Parquet still the best columnar file format in 2026?

For general interoperability it remains the default, because nearly every engine and table format reads it. For specialized workloads it is no longer automatically best: random row access, large embeddings and CPU-bound scans are where Lance and Vortex were designed to improve. The Parquet project is evolving too, with ALP in preview and further encodings and a new footer reportedly in progress.

What is the Lance format and when should I use it?

Lance is a columnar file and dataset format that picks mini-block or full-zip structural encodings per column to deliver fast random access while keeping scans competitive. Use it for ML training data, vector search and tables mixing scalars with large binary or embedding columns. For pure SQL scan workloads shared across many engines, Parquet is usually the lower-risk choice.

What is the Vortex file format?

Vortex is an open-source extensible columnar format and compression framework, contributed by SpiralDB to the LF AI & Data Foundation in August 2025 as an incubation project. It uses pluggable and cascading encodings and aims to let engines operate on compressed data. Its authors claim large speedups over Parquet, but those figures come without published methodology on the announcement page, so test on your data.

Which Parquet encodings matter most for compression?

Dictionary encoding with RLE and bit-packing carries most of the benefit for low-cardinality columns. DELTA_BINARY_PACKED helps sorted or incrementing integers, DELTA_BYTE_ARRAY helps strings with shared prefixes, and BYTE_STREAM_SPLIT helps floating-point data when paired with a codec like ZSTD. ALP for floats is in preview in the spec. Actual use depends on the writer’s version and configuration, so inspect your files.

How do row group size and file size affect Parquet performance?

Row groups set the granularity of statistics pruning, parallel splits and writer memory use. Too small and you pay in footer size, per-chunk dictionary rebuilds and weak statistics; too large and writers exhaust memory and readers lose parallelism. Many teams target row groups of tens to low hundreds of MiB within files of several hundred MiB or more, but you should measure against your engine and storage.