DuckLake vs Iceberg: Catalog-as-Metadata Architecture Compared

Every Iceberg commit is a small construction project. To add one second of sensor readings, a writer creates a data file, a manifest, a manifest list and a new table metadata file, then asks a catalog to swap a pointer. Do that once a second for a year and you have 31 million snapshots and a maintenance problem larger than the data.

DuckLake vs Iceberg is the sharpest version of a question the lakehouse world has been circling: if a catalog database is already mandatory for consistency, why keep the rest of the metadata in files? DuckLake, which DuckDB Labs declared production-ready as v1.0 on April 13, 2026, moves all table metadata into ordinary SQL tables and leaves only Parquet on object storage.

This post is written as an architecture decision record for teams ingesting frequent small writes, such as IoT and digital-twin telemetry. It separates what DuckLake’s own sources prove from what is marketing, works the commit math, and ends with a decision matrix.

What this covers: the design of DuckLake 1.0, the Iceberg commit path it is reacting to, a verified reading of the “926x” headline, catalog scale and governance, interoperability, failure modes, and a decision you can defend.

Context and Background

Open table formats exist to make a pile of Parquet files behave like a table: atomic commits, snapshot isolation, schema evolution and time travel. Apache Iceberg is the incumbent. Its specification describes immutable Avro manifests listing data files, a manifest list per snapshot, and a table metadata file that carries schema, partition specs and snapshot history. Writers commit optimistically by swapping the table’s metadata pointer from the base version to the new one, and retry if someone else got there first.

That design was built for a world where the only durable, shared substrate was object storage. Files are the source of truth, and the catalog is only a pointer. The consequence is that every change, however small, produces a fresh set of files, and readers must fetch and parse a chain of them before touching data. Iceberg has tuned this heavily. Version 3 added deletion vectors, row lineage, the variant type and default column values, and Iceberg 1.11.0 in May 2026 is the current stable line per the community. Our own Iceberg v3 spec walkthrough covers those features in detail.

The unresolved cost is the commit path. As we argued in the Iceberg v4 vs v3 analysis, commit cost in v3 scales with table metadata size rather than change size. The v4 proposals, anchored on a root manifest that can inline small changes, address that directly. But as of mid-2026 the community’s own status writeups say no v4 release date exists and key questions such as partition tuple handling remain open. Nothing you can set on a table today gives you v4 behaviour.

DuckLake attacks the same problem from the opposite direction. Instead of making files smarter, it asks why they carry metadata at all. The DuckLake manifesto argues that Iceberg and Delta Lake already require a transactional database to provide catalog consistency, yet still encode snapshots and statistics into “a maze of JSON and Avro files”. DuckLake collapses the two layers: one SQL database holds every piece of metadata, and data stays in open Parquet.

For IoT and digital-twin workloads this matters more than in classic batch analytics. Telemetry arrives as many tiny appends from many devices, queries want recent data fast, and long retention makes snapshot and file counts explode. That is exactly the regime where per-commit file overhead dominates. It is also where a single-node DuckDB deployment is already attractive, as we showed in DuckDB 2.0 versus 1.5 benchmarks for IIoT.

Two disclosures before going on. First, DuckLake is young: the 1.0 tag is under five months old at the time of writing, while Iceberg has years of multi-vendor production history. Second, the DuckLake team authored most of the performance evidence, so I flag every number as vendor-published or illustrative.

The Core Argument: Metadata Belongs in a Database

In short: DuckLake keeps all table metadata, including snapshots, schema, file lists and column statistics, in a SQL catalog database such as PostgreSQL, SQLite or DuckDB, while data remains Parquet on object storage. A commit becomes one small database transaction instead of a chain of new metadata files, and query planning becomes a single SQL query.

The architecture in one picture

Figure 1: DuckLake’s two-part architecture. Every metadata concept lives in the catalog database; object storage holds only immutable data and delete files.

The specification requires only two building blocks. The first is a catalog database that supports transactions and primary key constraints as defined by SQL-92. The second is data storage, meaning Parquet files on object, block or file storage. The spec documents 28 catalog tables, including ducklake_snapshot, ducklake_snapshot_changes, ducklake_data_file, ducklake_delete_file, ducklake_column, ducklake_table_stats and ducklake_partition_column.

Because the catalog is relational, features that need separate files in Iceberg become rows. A snapshot is a row. A schema change is rows in the column table, keyed by a stable field identifier. A file’s min and max statistics are rows that the database can index and filter. The reference implementation supports DuckDB, SQLite and PostgreSQL as catalogs, and the docs list MySQL 8 as possible but with “a number of known issues”.

The manifesto’s design pitch has three parts: simplicity, scalability and speed. Simplicity is the “all just SQL” claim. Scalability is the separation of storage, compute and metadata, mirroring how BigQuery is described as using Spanner and Snowflake FoundationDB for metadata. Speed is fewer round trips and fewer small files.

The commit is a SQL transaction

Compare the two commit paths side by side. In Iceberg, the writer’s sequence involves at least four object writes before the catalog is even contacted for the swap.

Figure 2: The Iceberg commit path against a REST catalog. Four object PUTs precede the atomic pointer swap, and a 409 conflict forces manifest work to be redone.

In DuckLake the writer first uploads the Parquet file, unless the write is small enough to inline, then runs one transaction against the catalog. The manifesto’s own example commits a data-file row, table statistics rows, column statistics rows, a ducklake_snapshot row and a ducklake_snapshot_changes row inside a single BEGIN ... COMMIT.

Figure 3: The DuckLake commit path. Metadata work happens in one database transaction; conflict handling is a retry against the catalog, not a rewrite of files.

The cost of that transaction is roughly constant regardless of how many rows the data file holds, which is the manifesto’s point. The critical section, where conflicts can occur, shrinks to a database commit.

Conflicts are detected by a primary key

DuckLake snapshot IDs are sequential integers starting at zero, and ducklake_snapshot enforces a primary key on snapshot_id. If two writers both try to commit snapshot N+1, the database rejects one. The loser then reads ducklake_snapshot_changes to see what the winner changed.

If there are no logical conflicts, the transaction is retried in the catalog without rewriting any data files. Logical conflicts abort with an error. The documented categories are schema operations (duplicate creates, drops of modified schemas), table and view name collisions or operations on dropped objects, inserts into dropped tables, concurrent deletes from the same data file, and compaction of tables that were deleted from or dropped. Concurrent deletes from different data files of one table are permitted.

The docs do not describe configurable retry settings, so treat retry behaviour as fixed in the current release. Compare this with Iceberg, where a 409 typically means the client refreshes metadata, re-evaluates its requirements and often rewrites manifests before retrying. Iceberg’s retry is more expensive by construction, though its conflict rules are also well understood after years of production use.

Data inlining removes the small-file problem at the source

The feature that matters most to telemetry is data inlining. When an insert affects fewer rows than data_inlining_row_limit, DuckLake writes the rows into catalog tables instead of creating a Parquet file. The 1.0 release extends this to deletes and updates and adds full ALTER TABLE support on inlined tables. The default threshold is 10 rows.

Inlined tables carry the original columns plus row_id, begin_snapshot and end_snapshot, so snapshot semantics still work: deleting an inlined row just sets end_snapshot. Deletes of rows already in Parquet create inlined deletion records keyed by file and row. A later call to ducklake_flush_inlined_data, or a full CHECKPOINT, materialises accumulated rows into a single Parquet file. When flushed data includes deletions, DuckLake writes a partial deletion file so time travel still resolves correctly.

Two constraints are worth knowing now. Non-DuckDB catalogs store nested types (struct, map, list) as VARCHAR when inlining, and inlining of the new variant type works only with a DuckDB catalog. Both come from the docs.

Deeper Analysis: What the Numbers Actually Say

Reading the “926x” headline carefully

The launch coverage repeated a striking claim: 926x faster queries and 105x faster ingestion than Iceberg. The primary source is DuckLake’s data inlining blog post, and the numbers are real but easy to misattribute.

The benchmark simulates autonomous-vehicle sensor data with 23 columns, using an Amazon RDS PostgreSQL 16.10 catalog on a c7g.2xlarge instance and an S3 bucket in the same region. It inserts 100 rows per second in batches of 10 rows, producing 300,000 rows across 30,000 batches over 50 minutes of simulated data.

Within DuckLake, comparing inlining off against on, the published results are insert 1,964 s versus 375 s (5.2x), aggregation 1,574 s versus 1.7 s (926x) and checkpointing 30 s versus 2.1 s (14.5x). So the 926x figure is DuckLake without inlining versus DuckLake with inlining. It is a measure of how badly thousands of tiny Parquet files hurt reads, not a head-to-head with Iceberg.

The Iceberg comparison in the same post used a smaller 10,000-row subset: insert 10.88 s versus 1,148.77 s (105x), aggregation 0.09 s versus 83.06 s (923x) and checkpointing 0.28 s versus 52.83 s (189x). The Register’s coverage attributes the 926x and 105x figures to DuckLake against Iceberg, blending the two comparisons.

Caveats a careful reader should hold onto. The workload is deliberately the worst case for file-based formats: ten-row commits at high frequency, and the post does not describe tuning or compacting the Iceberg run. The authors are the format’s designers. I did not find an independent reproduction. Treat the ratios as an upper bound for the pathological small-batch regime, not a typical result, and note the Iceberg run was narrowed to 10,000 rows, presumably because it was too slow to finish at full scale.

None of that makes the finding wrong. It shows that if your writer produces 10-row commits, a format that turns each commit into files will hurt. The right takeaway is the mechanism, which is well understood, not the multiplier.

Worked commit math for a telemetry table

Here is an illustrative model, not a benchmark. Suppose a gateway fleet appends one micro-batch per second to a single table. That is 86,400 commits a day.

On Iceberg (v1 to v3 metadata), each commit writes at least four objects: data file, manifest, manifest list and metadata file. That gives about 345,600 object PUTs a day per table before any compaction. At an assumed illustrative price of $0.005 per 1,000 PUTs, that is about $1.73 a day, or roughly $630 a year, for one table’s metadata churn alone. The dollar figure is small; the operational figure is not. You accumulate 86,400 snapshots a day, so snapshot expiry, manifest rewriting and orphan-file cleanup become a continuous job. Commit cost also grows with metadata size, per the v3 analysis above.

On DuckLake with a Parquet write per batch, each commit is one PUT plus one catalog transaction: 86,400 PUTs a day and 86,400 small transactions. That is a quarter of the PUT volume. Your metadata growth is rows in Postgres, not files in S3. If batches are under the inlining limit, the object PUT disappears entirely for most commits and a periodic flush writes one file per interval, for example one file per hour instead of 3,600.

The catch is that a one-second batch from a busy fleet is rarely 10 rows. If a gateway sends 500 readings per batch, the default inlining threshold does not engage. You can raise data_inlining_row_limit, and that is exactly what an IoT team should test, but each inlined row lives in your catalog database until flushed. Size the catalog for that, which is the theme of the next section.

Catalog scale: where the SQL bet can lose

The manifesto claims that a PostgreSQL-backed DuckLake can already scale to hundreds of terabytes and thousands of compute nodes, and that one thousand nodes appending at one-second intervals works fine. The FAQ says the only limitation is the catalog database’s performance, and that with a relatively slow catalog you can still have terabytes and millions of snapshots. These are design claims by the authors, not third-party benchmarks.

The reasoning is sound. Metadata volume is orders of magnitude smaller than data volume, and PostgreSQL handles thousands of transactions per second. But the catalog is now a stateful service on the critical path of every read and write. In Iceberg, losing the catalog means you cannot commit; existing metadata files still sit in object storage. In DuckLake, losing the catalog database without backups means losing the table definitions, statistics and snapshot history, while orphaned Parquet files remain but are effectively unreadable as tables. Your backup and high-availability story is Postgres’s story.

There is a real design discussion on the DuckLake repository about exporting an Iceberg or Delta metadata snapshot so other clients can read tables independently of the SQL host, and as a hedge against corrupted metadata. As of my check it had no maintainer response, and a community extension called delta_export appeared in March 2026 to fill the gap. Whether official export lands is unknown.

Feature Walk-through: What DuckLake 1.0 Actually Ships

A format is only useful if the boring features work. Here is what the documentation confirms, grouped by the concerns an IoT platform team cares about.

Snapshots, time travel and schema evolution

Every commit creates a snapshot, and DuckLake keeps a record of all historic snapshots and their changesets until you expire them. You query history with SELECT * FROM tbl AT (VERSION => 3) or AT (TIMESTAMP => now() - INTERVAL '1 week'), and you can attach a whole database at a point in time with SNAPSHOT_VERSION or SNAPSHOT_TIME. A snapshots function lists what is available.

Because snapshots are rows, the manifesto says DuckLake can manage millions of snapshots, and a snapshot can reference part of a Parquet file. That last detail is why compaction is less urgent than in Iceberg: several small logical writes can share one physical file without losing their identity in the history.

Schema evolution uses stable field identifiers stored as column_id. You can add columns (with defaults), drop columns, rename columns, add or drop struct fields, and promote types losslessly, for example int8 to int16 to int32 to int64 or float32 to float64. Readers resolve files by field ID, cast on promotion, ignore dropped columns and fill defaults for files written before a column existed. This is the same field-ID idea Iceberg uses, and it is what makes rename and drop metadata-only.

Partitioning, sorting and file pruning

Partitioning is declared with ALTER TABLE tbl SET PARTITIONED BY (...) and supports identity, bucket, year, month, day and hour transforms. Partitioning affects only newly written data; older files keep their layout, which gives you partition evolution. Partition keys live in metadata rather than being encoded in file paths, though Hive-style paths are used by default and can be disabled.

Version 1.0 added bucket partitioning using murmur3 hashing, chosen so bucket assignments line up with Iceberg’s bucket transform. That is a deliberate interoperability decision: a DuckLake table bucketed by device ID produces the same bucket numbers Iceberg would compute.

Sorted tables are also new in 1.0. SET SORTED BY accepts column names or arbitrary SQL expressions, and data is sorted during compaction. Inserts can be sorted automatically as well, and sort_on_insert disables that. For telemetry, sorting by (device_id, ts) tightens row-group min/max ranges so pruning skips more data. The catalog stores file-level statistics in tables, so the planner can select files with one indexed query rather than scanning manifests.

Deletes, variant and geometry

Row-level deletes use positional delete files in Parquet, which the manifesto says are compatible with Iceberg. Version 1.0 also adds experimental support for Iceberg v3 deletion vectors stored as Puffin files with roaring bitmaps. If you have read our Iceberg v3 features post, this is the same mechanism, and its purpose here is to make delete data byte-identical whether written through Iceberg or DuckLake.

The new variant type is a binary-encoded semi-structured type that supports more types than JSON, including dates and timestamps, and can be shredded into typed columns. It is useful for heterogeneous device payloads. Remember the inlining caveat: variant inlining works only on a DuckDB catalog, and the roadmap lists variant inlining for non-native catalogs as a v1.1 item. Geometry types store bounding-box statistics per file so spatial predicates prune files, and can nest inside structs, lists and maps.

Encryption

DuckLake supports optional encryption of every data file written to the data store, enabled by passing ENCRYPTED when attaching. It uses Parquet encryption, with a new key generated for each write operation, so each file has its own key. The keys are stored in the catalog’s ducklake_data_file table in an encryption_key field and are handed to authorised readers at query time.

That has a clear security consequence. Whoever can read the catalog can decrypt the data, and whoever can read only the bucket cannot. The manifesto frames this as enabling zero-trust data hosting: the bucket can sit with an untrusted party while the catalog stays in your control. It also means catalog access control is your data access control, so a Postgres role compromise is a data compromise.

Maintenance: compaction, expiry and cleanup

DuckLake does not delete old data files on its own. The recommended routine is a sequence: flush inlined data, merge adjacent files with merge_adjacent_files, expire snapshots, clean up files no longer referenced, and rewrite files that carry many deletes. A single CHECKPOINT statement runs the full sequence. On the catalog itself, you also run VACUUM on PostgreSQL.

This is close to Iceberg’s maintenance model, with one important difference. In Iceberg the cleanup jobs are separate Spark or engine procedures that reason about manifests. In DuckLake they are SQL functions over catalog rows, and file deletion is a follow-on step after snapshot expiry. The operational burden does not disappear; it becomes simpler to reason about and less frequent when inlining is doing its job.

Governance, Multi-Engine Reach and the Catalog Question

Iceberg’s catalog ecosystem is the main asset

The strongest argument for Iceberg is not the file layout. It is the catalog protocol. The Iceberg REST catalog specification gives every engine one way to list tables, load metadata and commit changes, and it includes a transactions endpoint for atomic multi-table commits. Implementations include Apache Polaris, which announced graduation to a top-level Apache project on February 19, 2026, Project Nessie, Unity Catalog and cloud-provider catalogs. Our comparison of Polaris, Nessie and Unity covers how those differ on RBAC, credential vending and versioning.

That ecosystem delivers things a SQL catalog does not give you for free: vended, short-lived storage credentials, table-level RBAC integrated with identity providers, and a protocol that Snowflake, Databricks, Trino, Spark, Flink and others already speak. If the buyer of your data platform is a group of teams on different engines, that reach is decisive.

DuckLake’s reach is real but narrower

The v1.0 announcement lists eight documented integrations: Apache DataFusion (by Hotdata), Apache Spark (from MotherDuck), two Trino implementations and a Pandas dataframe client, alongside first-class DuckDB support. The ducklake extension is reported among DuckDB’s top ten core extensions by downloads, and production deployments exist at “dozens of companies”. Those are the project’s own figures.

Note what that means architecturally. Each engine needs a DuckLake connector that talks SQL to the catalog and reads Parquet itself. The specification is deliberately small enough that a dataframe client could be written against it quickly, and the project published a post on exactly that in May 2026. But there is no equivalent of the Iceberg REST protocol that a vendor warehouse can natively mount, and no Snowflake or Databricks first-party read path that I could verify.

Governance is the other gap. The DuckLake roadmap for v2.0, with no stated timeline, lists permission-based role management, Git-like branching and incremental materialized views. Today, access control is whatever your catalog database and object store provide. You can build row-level security in Postgres for the metadata, but there is no standard, engine-neutral policy layer, and no credential vending. The FAQ positions DuckLake as similar to Delta Lake with Unity Catalog or Iceberg with Lakekeeper or Polaris, which is accurate only for the storage and metadata role, not for the governance features those catalogs add.

Interoperability: copy paths, not a shared protocol

DuckLake 0.3 introduced Iceberg interoperability via DuckDB’s iceberg extension. You can import from Iceberg to DuckLake as a deep copy of all data, or as a metadata-only copy using iceberg_to_ducklake() that preserves snapshot history and lets you time travel old Iceberg snapshots as DuckLake versions. Going the other way, you can copy data from DuckLake into existing Iceberg tables, but the system does not create or replace the Iceberg tables for you.

Because data files and positional delete files are Iceberg-compatible, a metadata-only migration is possible in principle. In practice, treat the current state as one-directional convenience: import is well supported, while live export of Iceberg metadata from a DuckLake table so that Trino or Snowflake could read it in place is not an official feature I could confirm. If multi-engine read access to the same live table is a hard requirement, plan on copying or on running Iceberg as the system of record.

Trade-offs, Gotchas, and What Goes Wrong

The catalog is now a tier-one service. DuckLake makes the SQL database the single point of consistency and the single point of failure. You need Postgres high availability, point-in-time recovery and tested restores. Losing catalog data is worse than losing an Iceberg REST catalog, because the metadata in Iceberg still exists as files in the bucket.

Inlining moves the small-file problem, it does not delete it. Rows inlined into Postgres consume catalog storage, WAL and vacuum capacity. A long-running telemetry firehose with a high inlining threshold can bloat the catalog. Flush on a schedule and monitor table sizes for the inlined tables.

Cross-catalog writers are a conflict hotspot. Any format with one commit point serialises writers per catalog. Snapshot IDs are sequential across the whole catalog, so in my reading of the spec a busy multi-tenant catalog will see more snapshot key conflicts than Iceberg’s per-table optimistic model does. The retry is cheap because it needs no file rewrites, but tail latency at high writer counts is unpublished.

Non-DuckDB catalogs lose type fidelity when inlining. Nested types become VARCHAR strings, and variant inlining only works on a DuckDB catalog. If your payloads are nested JSON-like structures and your catalog is Postgres, test round trips before committing.

MySQL is a trap. The docs explicitly warn about known issues. A DuckDB file catalog allows one client at a time, and SQLite supports multiple local clients with a single writer. Only PostgreSQL is a multi-user answer.

The evidence base is thin and vendor-authored. Version 1.0 promises backward compatibility, which is valuable, but the project has one primary engine and one main sponsor. Independent benchmarks at scale, failure-recovery reports and long-term support commitments are not yet published. A Snowflake principal engineer quoted by The Register expressed skepticism that a fledgling format could displace entrenched Iceberg and Delta ecosystems. That is a market argument, not a technical one, but it counts for procurement.

Iceberg’s answer is coming but not here. If you chose DuckLake mainly to escape per-commit file cost, remember that the v4 root manifest proposal targets the same issue inside Iceberg. If it ships and engines adopt it, the gap narrows, and your choice becomes about catalogs and governance. There is no confirmed v4 release date.

Decision Record: Choosing a Table Format for Small, Frequent Writes

Context

A platform team ingests telemetry from thousands of devices via MQTT and Kafka, lands it in object storage, and serves both dashboards and ad hoc analytics. Commits are frequent and small, retention is long, and a few downstream consumers use engines other than DuckDB. The team must pick a table format and catalog for the next three years.

Options considered

Option A is Iceberg with a REST catalog such as Polaris, Nessie or Unity, writing through Flink or Spark and compacting on a schedule. Option B is DuckLake 1.0 with a PostgreSQL catalog, writing through DuckDB workers and using data inlining. Option C is hybrid: DuckLake for the hot, high-frequency tier, with periodic copy into Iceberg for shared, governed consumption.

Figure 4: A decision flow for telemetry lakehouses. Engine reach and governance push toward Iceberg; commit rate and operational simplicity push toward DuckLake.

Decision matrix

The scores below are my qualitative judgement from the sources cited, not measurements. “Strong” means the documented design clearly serves the criterion; “Adequate” means it works with effort; “Weak” means a gap I could verify.

Criterion	Iceberg with REST catalog	DuckLake 1.0 with Postgres	Notes
Small frequent commits	Adequate, needs batching and compaction	Strong, inlining and single transaction	Vendor-published benchmark, worst-case workload
Commit conflict cost	Adequate, may rewrite manifests on retry	Strong, metadata-only retry	Tail latency at high writer counts unpublished
Metadata scale	Adequate, depends on snapshot expiry	Strong to unknown, bounded by catalog DB	Claims are authors’ design claims
Multi-engine reach	Strong, broad REST support	Adequate, DuckDB first-class, few connectors	Eight documented integrations at 1.0
Governance and RBAC	Strong, Polaris and Unity offer RBAC and credential vending	Weak to adequate, DB roles only, v2.0 roadmap	No engine-neutral policy layer today
Encryption	Adequate, storage-level or engine-level	Strong, per-file Parquet encryption keyed in catalog	Catalog access equals data access
Time travel and schema evolution	Strong	Strong	Comparable feature sets
Operational simplicity	Adequate, catalog service plus maintenance jobs	Strong, one Postgres plus SQL maintenance	Postgres HA is your responsibility
Ecosystem maturity	Strong, multi-vendor, years of production	Adequate, 1.0 in April 2026	Backward-compatibility guarantee helps
Format portability	Strong, files are the source of truth	Adequate, metadata trapped in the DB	Import from Iceberg supported, live export not official

Decision

For a single-team, DuckDB-centric telemetry platform whose consumers are mostly dashboards and notebooks, choose DuckLake with a PostgreSQL catalog. The simplification is real: one database, SQL maintenance, and inlining absorb the pathology that makes Iceberg streaming ingestion expensive.

For a platform that serves many engines or business units, or that must integrate with an existing vendor catalog and enterprise RBAC, choose Iceberg, write in larger batches, and plan for the v4 changes rather than wait for them. For a mixed situation, Option C is defensible, but it doubles the number of systems you operate.

Consequences

Choosing DuckLake means owning Postgres high availability and backups as part of the data platform. It means accepting a smaller connector set and building your own access policy. It also means writing a flush and checkpoint schedule, monitoring inlined-table growth, and keeping an exit path through the Iceberg copy tools.

Choosing Iceberg means investing in a batching layer, whether Flink checkpoint intervals or a Kafka-to-Iceberg sink, and continuous snapshot expiry and compaction. It buys you the widest reach and the most mature governance.

Practical Recommendations

Start with the write pattern, not the brand. Measure your commit rate and rows per commit for a representative week. If most commits carry fewer than a few hundred rows and you cannot batch upstream, the file-per-commit cost of Iceberg is a real tax and DuckLake deserves a proof of concept. If you can batch to one commit per minute or larger, Iceberg’s overhead is manageable and its ecosystem advantage dominates.

Run a two-week pilot with production-shaped data rather than trusting any published multiplier. The published 926x figure is inlining on versus off inside DuckLake, and the Iceberg comparison used a 10,000-row subset. Your result will depend on batch size, catalog instance, and how well the Iceberg side was tuned.

Treat the catalog as production infrastructure from day one. Use managed PostgreSQL with automated backups, cross-zone failover and point-in-time recovery, and rehearse a restore. Keep the DuckLake catalog in the same region as compute and storage so that the single planning query and commit transaction stay in the millisecond range.

Plan your exit before you enter. Keep data files in standard locations, schedule a periodic Iceberg copy of tables that other engines need, and record the DuckLake spec version in your runbook.

A short checklist:

Measure commits per day and rows per commit before choosing.
Set data_inlining_row_limit deliberately and flush on a schedule.
Declare partitioning (for example day(ts) plus bucket(N, device_id)) and a sort key such as (device_id, ts).
Schedule CHECKPOINT and monitor snapshot count, inlined-table size and catalog VACUUM.
Enable ENCRYPTED when the bucket is shared, and lock down catalog roles.
Require an independent reproduction of any benchmark you rely on.
Revisit the decision when Iceberg v4 has a release date and again at DuckLake v1.1.

For the Iceberg side of the pipeline, our Iceberg production deep dive covers compaction and snapshot expiry in practice, and the catalog comparison helps you choose the REST implementation.

Frequently Asked Questions

What is DuckLake and how is it different from Iceberg?

DuckLake is an open lakehouse table format from DuckDB Labs that stores all table metadata, such as snapshots, schemas, file lists and statistics, in a SQL database, while data stays in Parquet files on object storage. Iceberg stores metadata in files (metadata JSON, manifest lists and Avro manifests) and uses a catalog only to swap a pointer. DuckLake therefore commits with one database transaction and plans queries with one SQL query.

When was DuckLake 1.0 released?

DuckLake v1.0 was announced on April 13, 2026, together with DuckDB 1.5.2. The DuckDB team describes it as the first production-ready release with guaranteed backward compatibility. The specification is MIT licensed. The v1.0 release added sorted tables, bucket partitioning, variant and geometry types, full data inlining for inserts, deletes and updates, and experimental Iceberg v3 deletion vectors. A v1.1 is planned; v2.0 has no timeline.

Which databases can serve as a DuckLake catalog?

The reference implementation supports DuckDB, SQLite and PostgreSQL 12 or later, with MySQL 8 available but carrying documented known issues. A DuckDB file suits one client, SQLite suits several local clients with a single writer, and PostgreSQL is the choice for multi-user lakehouses. The specification itself only requires a database with transactions and primary key constraints per SQL-92, so other databases are possible.

Can other engines like Spark and Trino query DuckLake?

Yes, in part. The 1.0 announcement lists eight documented integrations: Apache DataFusion, Apache Spark, two Trino implementations and a Pandas client, plus first-class DuckDB support. Each is a connector that reads the SQL catalog and the Parquet files. There is no equivalent of Iceberg’s REST catalog protocol that warehouses like Snowflake or Databricks mount natively, so reach is narrower today than Iceberg’s.

Is DuckLake really 926 times faster than Iceberg?

Not as commonly stated. The 926x figure in DuckLake’s data inlining post compares DuckLake with inlining against DuckLake without it, on a query workload made slow by 30,000 tiny Parquet files. The direct Iceberg comparison used a 10,000-row subset and showed 105x faster inserts and 923x faster aggregations. It is a vendor benchmark of a worst-case small-batch workload, so treat it as an upper bound.

Should I migrate my Iceberg tables to DuckLake?

Only if small frequent commits are your measured pain and your consumers can live with DuckLake’s connector set and governance model. DuckLake can import Iceberg tables as a deep copy or as a metadata-only copy that preserves snapshot history. But there is no official live export back, so a migration is close to one-way in practice. Pilot with a copy, and keep Iceberg where many engines need access.