Polars vs DuckDB vs DataFusion: Apache Arrow Analytical Engines Compared for 2026 Workloads

Polars vs DuckDB vs DataFusion: Apache Arrow Analytical Engines Compared for 2026 Workloads

Polars vs DuckDB vs DataFusion: Apache Arrow Analytical Engines Compared

Most “Polars vs DuckDB” arguments are really arguments about interface: do you want to chain method calls or write SQL? That framing hides the part that matters. Under the syntax, all three of these engines are columnar, vectorized, multi-threaded, and built around the Apache Arrow memory model. They differ in what they are: Polars is a dataframe library with its own optimizer and engines, DuckDB is an embedded database with a storage format and a transaction manager, and DataFusion is a query-engine toolkit you assemble other products from.

This matters now because 2026 changed the ground under all three. Polars vs DuckDB is no longer a question about a single-threaded pandas replacement: Polars 2.0 is in release candidate with the streaming engine as the default for lazy queries, DuckDB shipped 2.0.0 LTS on 2026-10-21, and DataFusion is on its 55.0.0 release with a thriving ecosystem built on top. You will leave with a mental model of each engine’s architecture, a decision matrix, interop patterns that avoid copies, and a checklist for choosing without running a misleading benchmark.

What this covers: the shared Arrow foundation, how each engine plans and executes a query, the lazy-evaluation optimizer in each, interop code, failure modes, and a practical recommendation by workload.

Context and Background

Ten years ago the single-node analytics stack was pandas plus NumPy: eager, mostly single-threaded, and bound by the memory of one process. Two developments broke that ceiling. The first was Apache Arrow, a language-independent columnar memory format that lets different systems share buffers without serialization. The second was the realization that a laptop or a single cloud VM now has dozens of cores and hundreds of gigabytes of RAM, so a well-built vectorized engine can handle datasets that once justified a cluster.

Three open-source projects now dominate that single-node space, and each came from a different lineage:

  • Polars began as a Rust dataframe library by Ritchie Vink, built on its own Arrow-compatible memory layout. It exposes a Python (and Rust, and others) expression API with both eager and lazy modes.
  • DuckDB came out of CWI Amsterdam as an in-process analytical SQL database, often described as “SQLite for analytics”. It is written in C++, ships as a single library with no dependencies, and has its own storage format as well as readers for Parquet, CSV, JSON and more.
  • Apache DataFusion is a Rust query engine inside the Apache Arrow project. It is deliberately not a finished database: it provides SQL and DataFrame APIs, a logical optimizer, physical operators, and extension points, and other systems embed it.

The trap is treating them as interchangeable. They overlap on the “run a group-by over a Parquet file” benchmark that dominates blog posts, but diverge sharply on everything around it: persistence, concurrency, extensibility, and who owns the plan. If you work on data pipelines, our look at dbt vs SQLMesh for transformation, CI testing and incremental models shows where an embedded engine like DuckDB commonly slots into transformation tooling, and the Iceberg vs Paimon table format comparison covers the storage layer all three read from.

A note on evidence. Vendor and project benchmarks in this space are real but narrow, and they age within months. Where this article cites a result it names the source and date; where it gives an example number it labels it illustrative. The recommendations rest on architecture, which changes slowly, rather than on leaderboard positions, which do not.

What “Arrow-native” actually means

The Arrow columnar format specifies how arrays of each type are laid out in contiguous buffers: a validity bitmap for nulls, a data buffer, and for variable-width types an offsets buffer. Because the layout is a standard, two libraries can agree that a buffer is an int64 column and read it in place. The Arrow C Data Interface and the C Stream Interface standardize how a pointer to such data crosses a library boundary, and that is what makes “zero-copy” between Python libraries possible.

“Arrow-native” is a spectrum, not a binary. DataFusion is built directly on the Rust arrow-rs crates and uses RecordBatch as its unit of work. Polars historically implemented its own Arrow-compatible columns and converts at the boundary with high efficiency, and it is worth knowing that Polars’ string handling uses a view-based layout that matches the Arrow “string view” type, which helps interchange. DuckDB uses its own internal vector format, optimized for its execution engine, and exports to and imports from Arrow through scanning functions and the Arrow interfaces. In practice all three exchange Arrow cheaply, but only one of them (DataFusion) computes directly on Arrow arrays throughout.

The Core Architecture: Three Engines, One Memory Model

The short answer: Polars, DuckDB and DataFusion all parse or build a logical plan, optimize it, and execute it with vectorized, multi-threaded operators over columnar batches. Polars is a dataframe library, DuckDB is an embedded database with persistent storage, and DataFusion is an embeddable engine toolkit. Their execution models differ in how they schedule work and how they handle data larger than memory.

Polars vs DuckDB vs DataFusion shared Apache Arrow architecture from API to execution to storage

Figure 1: The common pipeline of the three engines and where each one diverges in API surface, execution model, and storage.

Figure 1 shows the shared skeleton. A front end produces a logical plan; an optimizer rewrites it; a physical planner picks operators; an executor runs them over batches of columnar data; and readers pull from Parquet, CSV, or object storage. The differences are in the boxes at either end: what the user types, and what happens when the data does not fit in RAM.

Polars: a dataframe API over an optimizing query engine

Polars exposes expressions: composable, column-oriented descriptions of computation such as pl.col("temp").mean().over("device"). In the lazy API you build a LazyFrame, which records a logical plan rather than computing anything, and collect() triggers optimization and execution. This is the lazy evaluation query optimizer that separates Polars from pandas: because the whole query is visible before it runs, Polars can apply projection pushdown (read only the columns the query needs), predicate pushdown (filter at the scan), common-subplan and common-subexpression elimination, and constant folding. You can inspect the result with LazyFrame.explain().

Execution has historically had two engines. The default in-memory engine processes whole columns and relies on the data fitting in memory. The streaming engine processes data in chunks so that peak memory is bounded by the working set rather than the input size. According to the Polars 2.0 release-candidate announcement by Ritchie Vink, dated 2 September 2026, collect on a LazyFrame will use the streaming engine by default in 2.0. The in-memory engine remains available process-wide with pl.Config.set_engine_affinity("in-memory") or per query with collect(engine="in-memory"). As of that post, 2.0 is a release candidate (pip install polars==2.0rc1) with the final release expected in the following weeks; check the current state before depending on it.

That announcement also states an expectation that the streaming engine will be “easily 5x faster” in aggregate, and links to a benchmarks page. Treat the figure as the project’s own expectation on its own benchmark suite, not a guarantee for your workload.

DuckDB: a database that lives inside your process

DuckDB is a SQL-first system. A query goes through a parser, binder, logical planner, and a rule-based and cost-based optimizer that includes filter pushdown, join-order optimization, and statistics-driven decisions. It executes with a vectorized, push-based pipeline model: operators process fixed-size vectors of values at a time, and the engine parallelizes pipelines across threads using a morsel-style scheduler, as described in the DuckDB team’s published work on parallelism.

What makes DuckDB distinct is that it is a database. It has its own single-file storage format with ACID transactions through MVCC, a catalog, indexes, and extensions for things like Parquet, Iceberg, HTTP and S3 access, spatial data, and more. The 1.4 LTS line, announced in September 2025, added database encryption, the MERGE statement, and Iceberg write support, per the DuckDB 1.4.0 LTS announcement. DuckDB can also spill intermediate state to disk for larger-than-memory sorts, joins and aggregations, which is a core design property rather than a late addition.

On versions: the release listing on duckdb.org shows 1.5.0 on 2026-03-09, a series of 1.5.x patch releases through 1.5.6 on 2026-09-28, and DuckDB 2.0.0 LTS “Cinnamon Teal” on 2026-10-21. I have confirmed the existence and date of the 2.0.0 release from the release listing, but I did not read its announcement body, so this article does not describe 2.0 features. Read the official notes before upgrading.

DataFusion: the engine you build products on

DataFusion’s pitch is in its documentation: it is an extensible query engine, written in Rust, that uses Arrow as its in-memory format. It offers a SQL front end, a DataFrame API, a LogicalPlan and ExecutionPlan split, and a rich set of traits for adding your own table providers, functions, optimizer rules, and physical operators. Execution is asynchronous and multi-threaded on Tokio, and data streams between operators as RecordBatch streams.

It is a library first. You do not “install DataFusion” and get a database; you cargo add datafusion and build one, or use a system built on it. That is why the project describes itself in terms of ecosystem. InfluxData, which builds InfluxDB 3 on DataFusion, wrote about the “year of 1,000 DataFusion-based systems”, and the project maintains related subprojects: Ballista for distributed execution and Comet, a Spark accelerator. The project blog lists DataFusion 55.0.0 (August 2026), Comet 1.1.0 (October 2026) and Ballista 55.0.0 (October 2026) among its recent releases.

Once you see the pipeline, the usual comparison questions reorganize themselves. “Which is faster?” becomes “which optimizer rules fire on this query and how does the executor schedule it?”. “Which should I learn?” becomes “do I want to own a plan, own a database, or own an engine?”. The next sections go through the three mechanisms that actually decide outcomes: lazy planning, execution and memory model, and interop.

Deeper Analysis: Planning, Execution and Memory

Lazy evaluation and the optimizer: what each engine can see

A lazy evaluation query optimizer can only rewrite what it can see. That is the real asymmetry between eager and lazy code, and it applies to all three engines.

Consider a sensor-analytics query: read a year of Parquet files for 40 columns, keep three, filter to one site, group by device and hour, and compute a mean. In an eager dataframe workflow you read all 40 columns, filter, then select. In a lazy plan the optimizer pushes the column selection into the Parquet reader and the filter into the scan, so row groups whose min/max statistics exclude the site are skipped without being decoded. The saving is a function of your data layout (how well sorted or partitioned the file is by site) and the selectivity of the predicate; it is not a constant factor.

# Illustrative: Polars lazy plan, then inspect what the optimizer did
import polars as pl

q = (
    pl.scan_parquet("telemetry/year=2026/*.parquet")
      .filter(pl.col("site") == "plant-7")
      .group_by("device", pl.col("ts").dt.truncate("1h").alias("hour"))
      .agg(pl.col("temp").mean().alias("avg_temp"))
)
print(q.explain())          # optimized plan: projection and predicate at the scan
df = q.collect()            # streaming engine by default in 2.0 RC, per the announcement
-- Illustrative: the same query in DuckDB; EXPLAIN shows filter and projection pushdown
EXPLAIN
SELECT device, time_bucket(INTERVAL 1 HOUR, ts) AS hour, avg(temp) AS avg_temp
FROM read_parquet('telemetry/year=2026/*.parquet')
WHERE site = 'plant-7'
GROUP BY device, hour;

DataFusion offers the same through either its SQL or DataFrame API, with EXPLAIN showing the logical and physical plans. In all three, pushdown into Parquet relies on the file’s footer metadata, so the practical lever you control is file layout: sort or partition by your most common filter column, keep row groups reasonably sized, and write statistics.

The engines differ in optimizer maturity and philosophy. DuckDB’s optimizer is that of a database: it has table statistics, cardinality estimation and join-order optimization, because it manages its own tables and can maintain stats. Polars’ optimizer is largely rule-based, and the 2.0 announcement lists a cost-based planner and join reordering as planned future work, not current features. DataFusion has a logical optimizer with a rule set and a physical optimizer, and because it is extensible you can add rules; its statistics support depends on the table provider you plug in. A query with many joins is where these differences show up first.

Streaming, morsels and out-of-core behavior

Memory behavior is where “it worked on my sample” turns into an incident. The three engines take different approaches.

Polars vs DuckDB vs DataFusion execution models: morsel pipelines, streaming chunks, and Tokio partition streams

Figure 2: Execution models compared, from the source scan through parallel workers to the sink, with the point at which each engine can spill or must hold state.

Polars streaming processes the plan in chunks through a push-based pipeline, so a scan, filter and aggregate can run in memory bounded by the aggregation state rather than the input size. Operators that need all their input, such as a sort or the build side of a join, still need memory proportional to what they hold. The 2.0 announcement lists out-of-core streaming support among planned work, which tells you it is not something to assume yet. Plan capacity as if the join build side and group-by state must fit in RAM.

DuckDB uses a buffer manager with a configurable memory_limit and can spill to a temporary directory when large operators exceed it. That makes it the most forgiving of the three when a query’s intermediate state outgrows memory, though spilling is much slower than in-memory processing, so it is a safety net rather than a goal. Set memory_limit and temp_directory explicitly in containers; Its default limit is derived from host memory, which may not reflect a container limit in your environment (unverified for current versions), and out-of-memory kills in Kubernetes are a recurring surprise.

DataFusion executes physical plans as partitioned RecordBatch streams on Tokio. Its memory pool tracks allocations from operators, and a spill-capable operator can write to disk when the pool is exhausted; which operators spill and how well depends on version and configuration, so verify the operators in your query plan against the current documentation rather than assuming. Because DataFusion is a toolkit, products built on it make their own choices here.

A row order caveat that bites during migration

One of the most practical details in the Polars 2.0 announcement concerns row order. The streaming engine does not guarantee row order by default for operations like join, group_by and unpivot. To keep the observable order you pass maintain_order=True, for example maintain_order="left" on a join. This is not unique to Polars: SQL has always said that without ORDER BY row order is unspecified, and DuckDB and DataFusion can return rows in different orders from run to run under parallel execution. But dataframe users are used to implicit order, and tests that compared exact output frames will start failing, intermittently.

The robust fix is not to preserve order everywhere; it is to make order explicit where it carries meaning (a time-series window, a “first row per key”) and sort or compare as sets everywhere else. Order maintenance has a cost, because it forces coordination between parallel workers.

Interoperability: moving data without copying it

The best argument for Arrow is that you can mix engines. The common pattern is to use DuckDB’s SQL for a gnarly join over Parquet, hand the result to Polars for expression-heavy feature engineering, and write Parquet out again.

Zero-copy Arrow interop between Polars, DuckDB and DataFusion through the C Data Interface

Figure 3: Data handoff between engines through Arrow buffers and the C Data Interface, and the points where a copy or a type conversion still happens.

# Illustrative interop sketch; verify method names against your installed versions
import duckdb, polars as pl

lf = pl.scan_parquet("devices/*.parquet")
df = lf.collect()                       # a Polars DataFrame

# DuckDB can query a Polars DataFrame in scope via replacement scans (Arrow-backed)
rel = duckdb.sql("SELECT site, count(*) AS n FROM df GROUP BY site")
out = rel.pl()                          # back to Polars through Arrow

# DataFusion in Python consumes and produces Arrow tables
from datafusion import SessionContext
ctx = SessionContext()
ctx.register_parquet("devices", "devices/")
arrow_tbl = ctx.sql("SELECT site, avg(temp) FROM devices GROUP BY site").to_arrow_table()

“Zero-copy” has fine print. Fixed-width numeric columns usually cross the boundary without a copy. Strings, nested types, dictionaries, and timestamps with different unit or timezone conventions can need conversion. Chunked arrays (many record batches) may be concatenated, which costs memory. And a result that has been materialized in one engine is a materialized result: handing over a LazyFrame plan does not share an optimizer. Interop works on data, not on plans. If you want to keep the whole query visible to one optimizer, keep it in one engine for as long as possible and cross the boundary once.

A useful rule is to cross the boundary at the point where the data is smallest. Do the heavy filtering and aggregation in the engine that is best at it, then hand over the reduced result.

What the benchmarks do and do not tell you

Public numbers exist, and they are worth reading with care. The Apache DataFusion project, via an InfluxData-republished post by Andrew Lamb, reported that DataFusion 43.0.0 was the fastest engine for querying partitioned Parquet files on ClickBench’s “hot” run, on a c6a.4xlarge (16 vCPU, 32 GB RAM) against a 14 GB partitioned dataset, with results dated 16 November 2024. The same post notes DuckDB was slightly faster on the cold run, that the ranking is likely to be fleeting, and that the author has a stake in the project. That is a snapshot of a single benchmark from late 2024; engines have released many versions since.

Benchmark claim Source Scope and caveat
DataFusion 43.0.0 fastest on ClickBench partitioned Parquet, hot run DataFusion project / InfluxData, Nov 2024 One benchmark, one machine class, author affiliated with the project; DuckDB faster on cold run
Polars streaming “easily 5x faster” in aggregate Polars 2.0 RC post, Sept 2026 Project expectation, links to its own benchmark page; not independently verified here
DataFusion 34 to 43 gained over 30% Same DataFusion post Self-reported across versions

Generic methodology caveats apply. Hot-run results depend on the OS page cache; Parquet file layout changes outcomes more than engine choice for many queries; a single-query benchmark cannot show concurrency behavior; and each engine is tuned for different data shapes. Run your own representative workload, with cold and warm runs, at your real data size, on your real hardware. I deliberately do not present a head-to-head table of my own numbers, because I did not run one.

Choosing: A Decision Matrix by Workload

Architecture suggests a division of labor more than a ranking. The matrix below reflects design properties documented above, not benchmark outcomes.

Situation Polars DuckDB DataFusion
Python feature engineering with complex expressions and window logic Strongest fit: expression API is the product Workable in SQL, verbose for deep column logic Possible via Python bindings, less ergonomic
Ad-hoc SQL over Parquet, CSV and Iceberg on a laptop or VM Needs SQL context, less central Strongest fit: SQL-first, many extensions Fine via CLI or Python, fewer batteries
Persistent local analytical store with transactions No storage engine Strongest fit: single-file database, MVCC Not a database; you build or adopt one
Larger-than-memory joins and sorts Streaming helps scans and aggregates; out-of-core listed as planned Spill-to-disk is a core design feature Depends on operator and configuration
Building your own database, service or query product Not designed as an embeddable engine C++ library, embeddable but less of a toolkit Strongest fit: extensible traits, Rust-native
Distributed execution Not its model Not its model (single node) Ballista and Comet subprojects exist
Rust-native application Good, Rust API exists C API and bindings Native

The three archetypes in practice

The data scientist or ML engineer working in notebooks and Python pipelines usually gets the best ergonomics from Polars for transformation, with DuckDB alongside for SQL joins and for querying files without loading them. The two interoperate through Arrow, so you rarely need to choose exclusively.

The analytics or platform engineer who wants a local warehouse for dbt-style models, CI tests, or quick investigation of data in object storage tends to land on DuckDB, because it is a database: there is a file, a catalog, and SQL that your analysts already know. This is why it appears so often in transformation tooling and test setups.

The systems engineer building a data product such as a time-series database, a log engine, or a query service is DataFusion’s audience. You get a planner, optimizer and executor written in a memory-safe language and customize them. The cost is that you own the product surface: persistence, catalogs, authentication, and operational concerns. For an IoT platform, that distinction is relevant: our comparison of Postgres 18, TimescaleDB and ClickHouse for IoT covers the server-side databases, while the engines here are what you would use for edge analytics, batch feature computation, and embedded query layers next to them.

A worked capacity estimate (illustrative arithmetic)

Capacity planning for a lazy query needs a model of working-set size. Suppose, as an illustration, a telemetry table with 2 billion rows, of which your query touches an int64 timestamp, a 4-byte device id, and an 8-byte float64 value. The raw decoded size of those columns is about 2e9 × (8 + 4 + 8) = 40 GB. An in-memory engine that decodes all three columns fully needs roughly that much plus working buffers. A streaming group-by by device and hour needs memory for the groups: if you have 50,000 devices and 8,760 hours in a year, there are up to 438 million groups, and at even 24 bytes of state each that is about 10 GB. The input is 40 GB but the aggregation state, not the input, sets the floor.

The lesson: before blaming an engine, estimate the group cardinality and the join build side. This arithmetic is illustrative only, with assumed sizes; real overhead per group differs by engine, and compression of the on-disk Parquet is unrelated to decoded memory.

Capacity decision flow for single-node analytics: dataset size, state size, and when to move to distributed execution

Figure 4: A decision flow from working-set estimate to engine choice and, when single-node limits are hit, to distributed options.

Trade-offs, Gotchas, and What Goes Wrong

Version churn is real. All three projects are moving fast: Polars is on the verge of a 2.0 major release with removed APIs and stricter typing (the announcement lists errors for lossy is_in coercion, strict-mode errors for horizontal concatenation of different-height frames, and removed arguments that raise AttributeRemovedError or ArgumentRemovedError); DuckDB just shipped a 2.0.0 LTS; DataFusion publishes major-numbered releases roughly every couple of months. Pin versions, read changelogs, and run your test suite on upgrades. A pipeline that depended on a quiet coercion will break loudly, which is arguably better.

Implicit order assumptions. Covered above, but worth repeating as the top migration failure: if your output depends on order, state the order.

Lazy is not magic. A lazy plan only optimizes what it can express. User-defined Python functions (for example map_elements in Polars, or Python UDFs in any engine) are opaque to the optimizer, run row-at-a-time or batch-at-a-time through the interpreter, and usually block pushdown and parallelism. Prefer built-in expressions. If a UDF is unavoidable, isolate it after the data has been reduced.

Out-of-memory in containers. DuckDB’s memory_limit and Polars’ memory behavior are not automatically tied to your cgroup limit. Measure peak RSS on production-sized data, set limits explicitly, and leave headroom for the allocator and for Arrow buffers held by other libraries in the same process.

Concurrency and storage semantics differ. A DuckDB database file can be opened read-write by one process, or read-only by several; concurrent multi-process writers need a different architecture (check the current concurrency documentation, as this area evolves). If you need multi-user, multi-writer access, an embedded engine is the wrong tool, and a server database such as Postgres or ClickHouse is the right one.

Small data punishes you the other way. For files that fit comfortably in L3 cache-friendly working sets, thread startup and planning overhead can dominate. All three are fast enough that this rarely matters, but microbenchmarks on tiny inputs mislead.

Benchmarks decay. The DataFusion post itself calls the top ClickBench spot “likely fleeting.” Any ranking in an article, including this one, is a dated snapshot. Choose on architectural fit and on a test of your own workload.

Anti-pattern: using three engines for one query. Interop is cheap but not free, and each handoff materializes data and splits the optimizer’s view. Pick a primary engine per pipeline stage and cross boundaries deliberately.

Practical Recommendations

Start from the job, not the brand. If the work is expression-heavy transformation in Python, use Polars lazily and call explain() to confirm that pushdown occurred. If the work is SQL over files or a local analytical store, use DuckDB and give it an explicit memory limit and temp directory. If you are building a product that needs a query engine, evaluate DataFusion and budget for owning everything around it. Where the pieces meet, let Arrow carry the data and keep each query inside one optimizer as long as you can.

Before committing, test with your own data. Use representative file layouts, a realistic size, cold and warm runs, and the real concurrency pattern. Record the version of every engine alongside the result, because a number without a version is not reproducible.

A checklist to adopt:

  • Write down data size, group cardinality, and the largest join build side before choosing.
  • Lay out Parquet by your most common filter column, with statistics enabled and sensible row-group sizes.
  • Make row order explicit wherever it matters; assert on sorted or set-equal outputs in tests.
  • Prefer built-in expressions to UDFs; isolate unavoidable UDFs after data reduction.
  • Set memory limits and spill directories explicitly in containers; measure peak RSS.
  • Pin engine versions; run upgrade tests, especially across Polars 2.0 and DuckDB 2.0.
  • Use explain() or EXPLAIN in review for any query that touches a large table.
  • Cross engine boundaries once, at the smallest intermediate result.
  • Re-benchmark every six to twelve months; treat published rankings as dated.

Frequently Asked Questions

Is Polars faster than DuckDB?

It depends on the query, the data layout, and the versions. Both are vectorized, multi-threaded and columnar, and published benchmarks swing in either direction depending on workload and date. Polars is strongest for expression-heavy dataframe code; DuckDB is strongest for SQL and when joins or larger-than-memory work are involved. Benchmark your own workload at real size, with both cold and warm runs, and record the engine versions you tested.

Should I use DuckDB or Polars for a data pipeline?

Use whichever matches how your team expresses the logic. SQL-fluent teams, dbt-style models, and local analytical stores point to DuckDB; Python-first teams with intricate column logic point to Polars. Because both read and write Arrow and Parquet, they combine well: use DuckDB for joins and file queries, then Polars for feature engineering, and cross the boundary once.

What is the difference between DataFusion and DuckDB?

DuckDB is a finished embedded database with its own storage format, transactions, extensions and SQL dialect. DataFusion is a query engine toolkit written in Rust that uses Arrow in memory and is meant to be embedded in other products, with extension points for table providers, functions and optimizer rules. You use DuckDB; you build with DataFusion. Several databases, including InfluxDB 3, are built on DataFusion.

Does the Polars streaming engine handle data larger than memory?

Partly. The streaming engine processes data in chunks, so scans, filters and many aggregations run in bounded memory. Operators that must hold large state, such as big joins or sorts, still need that state to fit. The Polars 2.0 announcement lists out-of-core streaming support as planned work, so do not assume full spill-to-disk behavior. Estimate group cardinality and join build sizes, and test at production scale.

What is Apache Arrow’s role here?

Arrow is a standard columnar memory format and a set of C interfaces for passing buffers between libraries. It lets Polars, DuckDB and DataFusion exchange data with little or no copying for many types, and it underpins DataFusion’s execution directly. It does not make the engines identical: each has its own optimizer, scheduler and memory management, and interchange of plans is not part of Arrow.

Will Polars 2.0 break my existing code?

Possibly. The 2.0 release candidate changes the default collect engine to streaming, does not guarantee row order by default for joins, group-bys and unpivots, raises errors on lossy is_in coercion, and removes deprecated APIs such as melt in favor of unpivot. The official upgrade guide covers each change. Pin your version, run your tests on the release candidate, and add explicit maintain_order or sorting where order matters.

Further Reading

By Riju — about

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *