Iceberg vs Delta vs Hudi for Lakehouses: 2026 ADR

Iceberg vs Delta vs Hudi for Lakehouses: 2026 ADR

Iceberg vs Delta vs Hudi for Lakehouses: 2026 ADR

Last Updated: November 2026

If you are choosing a table format today, somebody in the room will say “just use Iceberg”, someone else will say “we are a Databricks shop, so Delta”, and a third person will ask whether Hudi is still alive. All three statements are partly right and none is a decision. The iceberg vs delta vs hudi question has stopped being a feature checklist, because the three projects have converged on deletion vectors, row-level lineage and catalog-mediated commits. What still differs is where the source of truth for a commit lives, who is allowed to write, how much metadata a reader must open before it reads a single row, and which vendor’s catalog you are quietly betting your next five years on.

This post is a working architecture decision record (ADR) you can adapt for your own docs/adr/ folder. It explains the mechanism behind each format, checks the claims that went stale since 2024, and ends with a decision flow and a migration plan. You will leave with a defensible default, a short list of conditions that overturn it, and the failure modes you accept by choosing it.

What this covers: context and current versions, the shared architecture and how each format departs from it, commit and concurrency mechanics, deletes and CDC, catalogs and interoperability, a decision matrix, gotchas, and an FAQ.

Context and Background

An open table format is a metadata layer over immutable columnar files, almost always Parquet, in object storage. It turns a directory into a table with atomic commits, snapshot isolation, schema evolution, time travel and row-level mutation. Without it you have a data lake: cheap and flexible, but hostile to updates, deletes and concurrent writers. With it you have a lakehouse, the same storage economics plus database-like guarantees.

Three projects matter in 2026. Apache Iceberg began at Netflix and is governed by the Apache Software Foundation. The project documentation currently lists 1.12.0 as the latest release, with 1.11.0 before it. Delta Lake began at Databricks and now lives under the Linux Foundation. Its GitHub releases page shows 4.4.0, built against Apache Spark 4.2.0, 4.1.0 and 4.0.1. Apache Hudi began at Uber and reached its 1.0 milestone in December 2024; I could not confirm a later numbered release from the sources I checked, so treat the Hudi version specifics below as 1.0-line facts.

Several things changed since the 2023 and 2024 comparisons that still dominate search results. First, the Iceberg v3 specification was closed with the 1.10 release, according to the Google Open Source blog on Iceberg 1.10. It brings deletion vectors, row lineage and a variant type for semi-structured data. Second, Delta Lake 4.0 added catalog-managed tables as a preview, plus the variant type, Delta Connect and a “drop feature” operation, as described in the Delta Lake 4.0 announcement. Third, the cloud providers made Iceberg a first-class managed object: Amazon S3 Tables documents Iceberg v3 support, including deletion vectors and row lineage, in its user guide.

The old framing, “three equivalent formats fighting a turf war”, is therefore wrong in a useful way. The formats now share a vocabulary, so the differences move to operations: commit protocol, catalog ownership, compaction economics and ecosystem gravity. For catalog specifics see our comparison of Iceberg catalogs: Polaris vs Nessie vs Unity, which pairs naturally with this record.

One claim from the stale version of this article deserves correction. It implied that choosing a format locks your data in. In practice the data files are Parquet and remain readable by anything. What you lock in is the metadata, the catalog binding and the write path, which are exactly the parts that are expensive to rewrite. That distinction drives most of the decision below.

The Shared Architecture and Where Each Format Departs

All three formats solve the same problem: object stores give you atomic single-object writes (and, on modern S3, conditional writes) but no multi-file transactions. A table commit must therefore be reduced to one atomic operation on a small piece of metadata. Everything else, the data files, can be written ahead of time and referenced only after the commit succeeds.

Direct answer: Iceberg tracks a table as a tree of immutable metadata files swapped atomically through a catalog pointer. Delta tracks it as an ordered log of JSON commits compacted into checkpoints. Hudi tracks it as a timeline of instants over file groups of base and log files, optimised for frequent record-level upserts.

iceberg vs delta vs hudi shared lakehouse architecture from engine to object storage

Figure 1: The common lakehouse stack. Engines talk to a catalog, the catalog resolves table metadata, and metadata references data and delete files in object storage.

Figure 1 shows the layers they share. An engine such as Spark, Flink or Trino asks a catalog where a table’s current metadata lives. The metadata lists manifests or log entries, which in turn list immutable Parquet files and, where row-level deletes exist, delete files or deletion vectors. The formats differ in how the middle layers are built, and that difference decides query planning cost and write contention.

Iceberg: a metadata tree with a catalog pointer

Iceberg represents a table as a metadata JSON file that points to a manifest list per snapshot. The manifest list points to manifest files, and each manifest lists data files with per-column statistics and partition values. A commit writes new manifests, a new manifest list and a new metadata JSON, then asks the catalog to swap the table’s pointer from the old metadata file to the new one. If the pointer has moved in the meantime, the swap fails and the writer must retry.

This design has two consequences. Query planning can prune at three levels: the manifest list (partition ranges per manifest), the manifest (file-level statistics), and the data file footer. A reader never lists a directory, which matters on object stores where listing is slow and, historically, eventually consistent. And because the catalog performs the compare-and-swap, correctness depends on the catalog implementation, not on object-store semantics. This is why the REST catalog specification matters so much: it standardises the commit endpoint for every engine.

Iceberg also separates the table spec (v1, v2, v3) from the library release. The v3 spec adds binary deletion vectors stored in Puffin files, row lineage fields, a variant type and additional types such as nanosecond timestamps, according to the AWS documentation on S3 Tables. Upgrading a table from v2 to v3 is one way: AWS states the table cannot be downgraded through standard operations.

Delta: an ordered log with checkpoints

Delta stores a _delta_log directory beside the data. Each commit is a numbered JSON file such as 00000000000000000042.json containing add and remove actions, metadata changes and protocol upgrades. Periodically the log is summarised into a Parquet checkpoint so readers do not replay history from version zero. A commit succeeds by atomically creating the next-numbered file, which requires put-if-absent semantics from the storage layer, or a coordinator when the store lacks them.

That coordinator problem is why the catalog-managed direction exists. The Delta 4.4.0 release notes describe a Delta Kernel committer for Unity Catalog and the “UC Delta API” for catalog-managed tables, including credential vending, and an experimental Flink connector uses it for table loads and commits. In effect, Delta is moving from “the file system is the source of truth” toward “the catalog arbitrates commits”, which is the Iceberg model viewed from the other side.

Delta’s evolution is governed by a protocol version plus named table features. A table declares which reader and writer features it needs, so an old client refuses to touch a table it cannot safely handle. The new drop-feature operation lets you remove a feature without erasing history, which eases the pain of one-way protocol upgrades.

Hudi: a timeline and file groups

Hudi starts from a different question: how do you make record-level upserts cheap? It organises data into file groups, each holding a base file plus append-only log files in Merge-on-Read (MoR) tables. A primary key maps each record to a file group through an index. A timeline records every action, such as commits, compactions and cleans, as instants with requested, inflight and completed states.

Hudi 1.0 redesigned the timeline as a log-structured merge (LSM) tree. The Hudi 1.0 announcement reports reading the timeline for 10 million instants in about 3.4 seconds, spread over 8.4 GB of files. Each action now carries a requested time and a completion time, which is what makes its new concurrency model possible.

The upshot is that Hudi pays an indexing and compaction cost on writes so that upserts and incremental reads stay cheap, while Iceberg and Delta started from analytic reads and added efficient mutation later. That historical bias still shows in how they behave under heavy change-data-capture (CDC) load.

Metadata layout comparison for iceberg vs delta vs hudi tables

Figure 2: Metadata layout. Iceberg walks a four-level tree, Delta reads a checkpoint plus recent JSON commits, and Hudi consults an LSM timeline to find file groups made of base and log files.

Figure 2 contrasts the read paths. Iceberg needs the catalog lookup plus up to four hops before it reads data, but each hop prunes aggressively. Delta needs the latest checkpoint and any commits since, which is cheap until the log tail grows long between checkpoints. Hudi consults the timeline and then merges base and log files for MoR reads, which is where compaction lag turns into query latency.

Why the layout matters for cost

Planning cost is paid on every query, not once. A table with millions of small files and a poorly maintained metadata tree makes every query slow, whichever format you chose. All three therefore need scheduled maintenance: compaction of small files, snapshot or log expiry, orphan file cleanup and, for MoR tables, delete-file compaction. The format that wins in your environment is often simply the one whose maintenance jobs your platform team already runs reliably.

Delete handling deserves its own mention, because it is where the three converged most. Iceberg v2 used positional and equality delete files. Reading a table with many of them means joining data files against delete files, which is expensive and produces small-file sprawl. The v3 binary deletion vector replaces positional delete files with a compact bitmap stored in a Puffin file, reducing write amplification and small files, per the AWS documentation. Delta has had deletion vectors for longer, and the 4.4.0 notes show Flink’s default upsert strategy using them to remove old row versions without rewriting data files. Hudi avoids the problem differently: log files hold the changes and compaction merges them.

Commit Protocols, Concurrency, and Failure Behavior

The deepest difference between the formats is how they behave when two writers collide. Reads are easy: every format gives snapshot isolation by pointing a reader at one immutable snapshot. Writes are where operational reality arrives.

Direct answer: Iceberg and Delta use optimistic concurrency control: writers prepare changes against a snapshot, attempt an atomic commit, and on conflict they validate and retry. Hudi 1.0 adds non-blocking concurrency control, letting multiple writers touch the same file group without one aborting the other, at the price of a more complex merge model.

Optimistic concurrency in Iceberg and Delta

Under optimistic concurrency, each writer reads version N, writes new data files, and attempts to commit version N+1. If another writer got there first, the commit fails. The writer then refreshes, checks whether its changes conflict with what landed, and either retries cheaply or aborts. Two appends rarely conflict. Two overlapping MERGE or UPDATE operations on the same files conflict almost always.

Optimistic commit sequence for iceberg vs delta vs hudi writers contending on a catalog

Figure 3: Two writers race to commit version N+1. The catalog accepts one and rejects the other, which must refresh, validate for logical conflicts and retry.

Figure 3 shows the sequence. The important detail is the retry cost. A retry after a pure metadata conflict is cheap because the data files are already written; a retry after a logical conflict means redoing the rewrite. This is why partition design is a concurrency design: if two jobs update disjoint partitions, their file sets do not overlap and validation passes.

Iceberg 1.10 tightened one failure case that bites in production. According to the Google post on the release, the client no longer retries a commit after a 5xx server error, because the failed request may in fact have succeeded and a blind retry could create conflicting operations or corrupt the table. The correct handling is to treat the outcome as unknown and reconcile, which is the same discipline you apply to any non-idempotent RPC. Our article on Kafka exactly-once semantics and idempotent producers covers the same principle for streams.

Non-blocking concurrency in Hudi

Hudi 1.0 introduced non-blocking concurrency control (NBCC). Per the release announcement, multiple writers can write to and compact the same record concurrently without blocking each other, using lightweight distributed locks and TrueTime-style time semantics (RFC-66). The announcement demonstrates two Flink jobs concurrently enriching different columns of the same table, and says support for other compute engines is planned.

The trade-off is that records from both writers land in log files, and ordering is resolved at read or compaction time using the completion-time semantics. That is powerful for streaming enrichment pipelines, where several jobs each own a column family. It is overkill, and harder to reason about, for a nightly batch with one writer.

Catalog-mediated commits and the lock problem

Each format needs an atomic arbiter. In Iceberg it is the catalog’s compare-and-swap. In Delta it was historically the storage layer’s put-if-absent behavior, with a lock service such as DynamoDB for S3 setups before S3 supported conditional writes; the move to catalog-managed commits replaces that. In Hudi it is the lock provider plus the timeline.

Which arbiter you trust is an operations decision. A REST catalog gives you a single, auditable commit endpoint, central access control and credential vending, and it makes multi-engine writes safe. A file-system-arbitrated model has fewer moving parts but gives you no place to enforce policy. In 2026 the trend in all three projects is toward the catalog arbiter, so if you pick a format whose catalog story you dislike, you are swimming against its roadmap.

Failure modes worth rehearsing

Several failure modes recur across formats. An orphaned data file appears when a writer dies after uploading files but before committing; it is harmless for correctness and only wastes storage until orphan cleanup runs. A stuck commit occurs when a catalog loses connectivity mid-swap; the writer cannot tell whether it succeeded, so your job must be idempotent or reconcile through snapshot summaries. Metadata bloat happens when snapshot expiry is never run and the table accumulates thousands of manifests or log files. Long-running readers pin old snapshots, and expiring those snapshots underneath them produces file-not-found errors.

None of these is format specific, but each format has a different default exposure. Test them in staging by killing a writer at each phase of a commit, rather than trusting the documentation. A one-day chaos exercise saves a quarter of production confusion.

Deeper Analysis: Deletes, CDC, Catalogs, and Interoperability

Row-level change and CDC economics

Most lakehouse pain comes from one workload: applying a stream of inserts, updates and deletes from an operational database. Parquet files are immutable, so every update has to either rewrite a file (copy-on-write) or record the change beside it (merge-on-read). The choice shifts cost between the writer and the reader.

Copy-on-write rewrites each touched file in full. Writes are expensive and read amplification is zero, which suits tables that are mostly read and updated in large, infrequent batches. Merge-on-read writes a small delta, either a delete file, a deletion vector or a log file, and defers the merge to read time or to a compaction job. Writes are cheap, reads pay a merge tax, and compaction becomes a mandatory operational process.

To see why this matters, take an illustrative example, not a benchmark. Imagine a table of 10,000 files of 256 MB each, and a nightly batch that updates 0.5 percent of rows scattered randomly. If the updates touch rows in nearly every file, copy-on-write rewrites close to the entire 2.5 TB table to change a few gigabytes of logical data. Merge-on-read writes only the changed rows plus a bitmap marking which old rows are dead. That gap, a large multiple in bytes written, is the whole argument for deletion vectors and Hudi’s log files. Real numbers depend on your key distribution, so measure on your own data.

Hudi publishes its own measurements for this problem. The 1.0 announcement reports that partial updates, stored in the log format so MERGE INTO writes only changed fields, improved Merge-on-Read query performance about 2.6 times and cut write amplification by 85 percent on a 1 TB benchmark, and in an 80 percent random-update test on a 1 TB MoR table reduced bytes written 70.2 times with 5.7 times better query latency. These are vendor-project benchmarks on specific workloads. Treat them as proof of the mechanism, not as a promise for your pipeline.

Iceberg’s answer is the v3 deletion vector plus row lineage. Deletion vectors avoid the read-modify-write cycle of rewriting positional delete files. Row lineage adds _row_id and _last_updated_sequence_number fields so downstream consumers can identify changed rows without diffing snapshots, which is useful for incremental processing and CDC, per the S3 Tables documentation. One caveat from that same page: upgrading does not backfill lineage for history, so incremental consumers can only rely on it for changes after the upgrade.

Delta already offers deletion vectors and row tracking, and the Delta 4.0 Kernel work added support for writing tables with deletion vectors and for row tracking and clustered tables. Delta also offers liquid clustering, which replaces rigid partitioning with an adaptive layout; Iceberg achieves a similar effect with sort orders and compaction strategies, and Hudi with clustering services. The practical lesson is that no format is incapable of upserts any more. They differ in how automatically the maintenance happens.

Indexing: where Hudi still stands apart

Hudi’s index is the feature the other two do not have an equal for. Record-level indexes and, from 1.0, secondary indexes let a writer locate a record’s file group without scanning. The release announcement states that Spark SQL can create and drop secondary indexes, that an asynchronous indexing service builds them without interrupting writers, and that on a 10 TB TPC-DS web_sales benchmark query latency dropped roughly 88 to 95 percent. It also states that only Spark supports them at that release, with Flink, Presto and Trino planned for 1.1.

That last constraint is the real caveat. A format feature that only one engine honors is a feature of that engine. Before you design around Hudi indexes, check the support matrix of every engine that will read the table, because a reader that ignores the index still returns correct results but loses the speedup.

Iceberg and Delta rely instead on min/max statistics, partition pruning, Bloom filters in Parquet and data layout. Those are excellent for analytic scans and weaker for point lookups by a high-cardinality key. If your workload is “find this one customer record among billions”, a lakehouse table is not the right primary store anyway; pair it with an operational database or a serving layer.

Catalogs and the interoperability question

The catalog is where the formats are won or lost in practice. Iceberg’s REST catalog specification gives any engine one commit and metadata API, with credential vending so engines receive short-lived storage credentials rather than long-lived keys. Google Cloud’s BigQuery metastore catalog support and a Google auth manager arrived in the 1.10 line, and AWS exposes an Iceberg REST endpoint for S3 Tables. The result is that “Iceberg” now means “a table format plus a standard way to find and commit to it”, which no competitor offered equally in 2024.

Delta’s equivalent is Unity Catalog and the open-source Unity Catalog project, with the 4.x line pushing catalog-managed tables. If your estate is centered on Databricks, this is a strength: governance, lineage and access control come built in. If it is not, you are depending on a catalog whose richest features live in one vendor’s product.

Interoperability layers try to close the gap. Delta UniForm writes Delta tables while generating Iceberg metadata alongside, so Iceberg readers can query the same Parquet files. The 4.4.0 notes mention atomic UniForm metadata initialization with table creation and a delta-iceberg module that supports Spark 4.1 but not Spark 4.2, a good example of the version-coupling you must track. Apache XTable takes a different approach, translating metadata between Hudi, Delta and Iceberg after the fact.

Be precise about what these layers give you. They give cross-format reads. They do not give you a second writer: with UniForm the Delta side remains the one writer, and Iceberg engines read a derived view. Metadata translation also adds latency between a Delta commit and the matching Iceberg metadata becoming visible, so a pipeline that needs read-your-writes across engines must account for it.

For governance and schema discipline that sits above all three formats, see our guide to data contracts, schema registry and lakehouse schema evolution, which covers where contract enforcement belongs relative to table metadata.

Engine and platform support in practice

Support claims such as “we support all three” hide a graded reality: read support, write support, row-level mutation support, and maintenance support are four different things. A short list of the facts I could verify from primary sources:

  • Amazon S3 Tables supports Iceberg v3 data types, a REST API and table maintenance for v3, but Amazon Athena cannot read v3 tables, and engine versions such as EMR Spark 7.12 or later and Glue 5.1 or later are required.
  • Delta 4.4.0 is built for three Spark lines, and its Iceberg compatibility module trails the newest Spark line.
  • Hudi’s secondary indexes work on Spark first, with other engines planned.
  • Iceberg’s v3 upgrade is one way, and V3 data types are Parquet only, not ORC or Avro.

The pattern is clear. New format features land in one engine first, and the cost of being early is a window in which some of your readers cannot open the table. Before upgrading any table spec or protocol version, inventory every reader, including BI tools and notebooks that nobody remembers owning.

A configuration sketch

The properties below show how each format opts into merge-on-read behavior. Iceberg’s property names come from the AWS v3 documentation; check your engine’s documentation for the rest, as names and defaults change between releases.

-- Iceberg: enable merge-on-read so v3 deletion vectors are used
ALTER TABLE lake.sales.orders SET TBLPROPERTIES (
  'format-version'    = '3',
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'write.merge.mode'  = 'merge-on-read'
);

-- Delta: expose Iceberg metadata via UniForm (verify property names for your version)
ALTER TABLE main.sales.orders SET TBLPROPERTIES (
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

-- Hudi: a merge-on-read table keyed on order_id
CREATE TABLE orders_hudi (order_id BIGINT, ts TIMESTAMP, amount DOUBLE)
USING hudi
TBLPROPERTIES (type = 'mor', primaryKey = 'order_id', preCombineField = 'ts');

Notice that the Hudi table declares a primary key and a precombine field up front, because Hudi must know how to identify and order records. Iceberg and Delta treat keys as a matter of how you write your MERGE condition. That difference is philosophical: Hudi models the table as a keyed dataset, the other two model it as a set of files.

Maintenance: the hidden operating cost

Every format needs a maintenance plane, and the cost of running it is the most under-quoted number in format comparisons. For Iceberg you schedule compaction (rewrite_data_files), manifest rewriting, snapshot expiry and orphan file removal, usually through your engine or a managed service. S3 Tables, for instance, documents managed table maintenance, including compaction, which shifts this burden to the provider at a price. For Delta you schedule OPTIMIZE, VACUUM and checkpointing, and Databricks can automate much of it. For Hudi you configure inline or asynchronous compaction, clustering and cleaning, and the timeline archival service.

Two questions determine your maintenance cost. First, who runs it: you, your platform team, or the vendor? Second, does it compete with writers for the same resources? Compaction jobs that rewrite files can themselves conflict with concurrent writers under optimistic concurrency, which is one reason Hudi’s non-blocking model appeals for always-on streaming tables. For a high-throughput streaming sink, budget for compaction throughput as a first-class capacity input, not an afterthought.

Time travel, branching, and reproducibility

All three support time travel by version or timestamp. Iceberg adds named branches and tags in the table metadata, which enables audit snapshots and write-audit-publish workflows without copying data. Delta offers restore and version-based reads, and Hudi exposes incremental and point-in-time queries driven by the timeline. For machine learning reproducibility, the question is whether you can pin a training set to an immutable snapshot identifier and still find those files months later. That depends on retention settings more than on the format: expiring snapshots to save cost also destroys reproducibility. Our comparison of data versioning for ML with lakeFS, DVC and Delta Lake discusses that trade-off in depth.

Decision matrix

The matrix below is my weighting for a net-new, multi-engine lakehouse. Scores are qualitative judgements for the stated assumptions, not measurements. Adjust the weights to your constraints.

Criterion Weight Iceberg Delta Hudi
Multi-engine read and write portability High Strong: REST catalog standard Good via UniForm reads, writes mostly Delta clients Moderate: Spark and Flink strongest
Streaming upsert and CDC at high rate High Good with v3 deletion vectors Good, deletion vectors, strong on Databricks Strongest: indexes, log files, NBCC
Managed cloud service availability Medium Very broad: S3 Tables, BigQuery, Snowflake Broad, richest on Databricks Narrower, Onehouse and cloud EMR/Glue
Governance and catalog features Medium Depends on chosen catalog Strong with Unity Catalog Depends on external catalog
Metadata scale and planning cost Medium Strong pruning, needs manifest care Strong, checkpoint tuning needed LSM timeline scales well
Operational simplicity Medium Moderate, catalog is a new component Simplest on Databricks Most tuning knobs
Ecosystem momentum for new projects Medium Highest High within one vendor orbit Smaller but focused

Decision flow for choosing between iceberg vs delta vs hudi

Figure 4: A first-pass decision flow. Multi-engine needs without a Databricks center of gravity point to Iceberg with a REST catalog; heavy CDC points to Hudi; Databricks-centric estates point to Delta with UniForm.

Figure 4 compresses the matrix into three questions. It is a starting point for the argument, not a replacement for it. The exceptions are the interesting part, and the next section covers them.

Trade-offs, Gotchas, and What Goes Wrong

Version coupling is the silent killer. Format specs, library releases, engine versions and catalog implementations all move independently. The Delta Iceberg-compatibility module trailing the newest Spark line, S3 Tables requiring specific EMR and Glue versions, and Athena being unable to read Iceberg v3 tables are all the same story. The mitigation is a compatibility matrix in your repository, tested in CI against every engine you rely on, before you upgrade any table version.

One-way upgrades. Iceberg v2 to v3 cannot be reversed through standard operations, per AWS. Delta protocol upgrades work similarly: enabling a feature raises the minimum reader and writer versions. Treat every upgrade as a migration with a rollback plan, which in practice means keeping a pre-upgrade clone or snapshot until all readers are verified.

Shredded variant surprises. On Iceberg v3, AWS notes that compaction writes shredded variant files by default and older readers may fail to read them; you can turn that off with write.variant.shredding.enabled=false. Delta likewise lists shredded variants as a preview feature. New semi-structured types are attractive, but they are exactly the features that break old clients first.

Small files and metadata bloat. Streaming writers commit frequently, and each commit creates metadata. Without compaction and expiry, planning time grows until the table becomes the bottleneck. Budget a compaction job for every streaming sink from day one.

Catalog as single point of failure. Once commits are catalog-mediated, an unavailable catalog stops all writes, and often reads. Run it with the same rigor as a database: replication, backups of its state, and a tested restore procedure. If you adopt a managed catalog, understand its service-level objectives and regional availability.

UniForm is not multi-writer. Teams sometimes assume a Delta table exposed as Iceberg can be written by Iceberg engines. It cannot; the Iceberg view is derived and read-oriented. If you need true multi-engine writes, you need a format with a catalog standard that all engines commit through.

Vendor-benchmark skepticism. Many published numbers, including the ones quoted above, come from project or vendor-run benchmarks on chosen workloads. They show a mechanism works, not that it will win on your data. Reproduce one representative workload, with your key distribution and your engines, before committing.

Hudi’s operational surface. Hudi gives the most tuning control and so asks for the most tuning knowledge: index types, table type, compaction scheduling, lock providers. Teams without a data-platform engineer who owns this tend to end up with a table that works until it does not. If you pick Hudi, staff for it.

Governance gaps. Row and column security, masking and lineage are catalog features, not format features. Moving from one catalog to another can silently drop policies. Export your access policies as code, so the catalog is replaceable.

Practical Recommendations

For a net-new lakehouse in late 2026 with no strong prior commitment, my default is Apache Iceberg behind a REST catalog, on spec v3 only once every reader you own can handle it. The reasoning is structural rather than fashionable: the format has the broadest managed-service support, a standard commit API that makes multi-engine access safe, and a catalog story that lets governance live outside any single compute vendor. This is an opinion, and the conditions below overturn it.

Choose Delta Lake if your platform is Databricks-centric. The integrated catalog, automated maintenance and optimizer features save real engineering time, and UniForm gives Iceberg-speaking engines read access. Do not choose it solely because it is popular, and do not expect UniForm to make your estate write-portable.

Choose Hudi if your dominant workload is high-rate upserts from CDC or streaming enrichment, especially with several concurrent writers on the same table, and you run Spark or Flink. Its indexes, partial updates and non-blocking concurrency were built for that. Confirm every reading engine supports the features you depend on.

Whichever you choose, apply the same operating discipline:

  • Pin a compatibility matrix of format version, engine versions and catalog version, and test upgrades in CI.
  • Schedule compaction, snapshot expiry and orphan cleanup before the first production write.
  • Decide snapshot retention explicitly, balancing storage cost against audit and ML reproducibility.
  • Keep the catalog highly available, with exported access policies and a tested restore.
  • Make writers idempotent so an unknown commit outcome can be reconciled safely.
  • Measure one representative workload on your own data before trusting any benchmark.
  • Document the exit path: Parquet files stay readable, but plan for metadata and catalog rewrite.

If you are migrating an existing table, do it incrementally. Snapshot or in-place metadata conversion tools exist for Iceberg, and cross-format translators such as Apache XTable can expose a second format while you move readers one at a time. Cut writers last. The reader-first order lets you roll back by pointing consumers at the old metadata.

Disclaimer: this article is a technical architecture analysis and not product, procurement or investment advice.

Frequently Asked Questions

Which is better in 2026, Iceberg, Delta or Hudi?

None wins universally. Iceberg is the safest default for multi-engine lakehouses because of its REST catalog standard and wide managed-service support. Delta Lake is strongest when your estate centers on Databricks and Unity Catalog. Hudi is strongest for high-rate upserts, concurrent writers and record-level indexing, mainly on Spark and Flink. Choose by writer topology, engine mix and who operates maintenance, then verify with a proof of concept on your own data.

Is Hudi dead?

No, but its position is narrower. Hudi 1.0 shipped in December 2024 with an LSM timeline, non-blocking concurrency control, secondary indexes and partial updates, and the project describes planned engine support in later releases. Its momentum in managed services is smaller than Iceberg’s, so check support in every engine you need. If your workload is heavy CDC on Spark or Flink, it remains a serious option.

Can I read a Delta table from Iceberg engines?

Yes, through Delta UniForm, which generates Iceberg metadata alongside the Delta log so Iceberg readers can query the same Parquet files. It is a read-oriented view: the Delta side stays the writer, and metadata translation adds latency. Version coupling applies too; the Delta 4.4.0 notes say the Iceberg module supports Spark 4.1 but not Spark 4.2. Apache XTable offers a related translation approach.

What does Iceberg v3 change?

The v3 spec, finalized with Iceberg 1.10, adds binary deletion vectors stored in Puffin files, row lineage fields, a variant type for semi-structured data and extra types such as nanosecond timestamps. Deletion vectors reduce write amplification versus v2 positional delete files. The upgrade is one way, and not every engine reads v3 yet; Athena, for example, cannot, per AWS documentation.

Do I still need a catalog if the format stores its own metadata?

Yes, for any serious deployment. The catalog arbitrates atomic commits, enforces access control, vends credentials and gives every engine one place to find tables. Iceberg depends on it for its pointer swap, and Delta is moving toward catalog-managed commits with Unity Catalog. Treat the catalog as a tier-one service with high availability and backups, and prefer a standard interface such as the Iceberg REST API.

How hard is migrating between formats?

The data files are Parquet and usually do not move, so the cost is metadata, catalog bindings, writers and maintenance jobs. Translation tools can expose a table in a second format during transition. The risky part is write-path cutover and governance policy, which live in the catalog. Migrate readers first, writers last, keep the old metadata until verified, and rehearse rollback on a clone.

Further Reading

By Riju — about

2 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *