Iceberg Catalogs: Polaris vs Nessie vs Unity Comparison (2026)

Pick the wrong table format and you rewrite some pipelines. Pick the wrong Iceberg catalog and you rewrite your security model, your multi-engine story and your disaster recovery plan, usually at the moment a second query engine arrives. The catalog is the one component every reader and writer must trust: it decides which metadata file is the current version of a table, who may read or commit to it, and which storage credentials an engine receives.

In October 2026 the choice is far clearer than it was two years ago, because the Iceberg REST catalog protocol turned the catalog into a swappable service. Apache Polaris has graduated to a top-level Apache project, Unity Catalog ships an open-source server, Nessie keeps its Git-style versioning niche, and AWS Glue speaks REST. The differences that remain are governance depth, branching, operations and lock-in.

This guide compares them on mechanisms rather than slogans, with a decision flow and a migration plan.

What this covers: what a catalog actually does, the REST protocol, Polaris, Nessie, Unity Catalog, Glue, Hive Metastore and Lakekeeper side by side, credential vending, multi-table commits, branching, governance, failure modes, migration and a checklist.

Last Updated: October 2026

Context and Background

Apache Iceberg separates a table into immutable files in object storage and a small mutable pointer. Data files, manifests and manifest lists never change after they are written. Each commit produces a new metadata JSON file describing the table as of that commit, and the catalog’s job is to atomically move the pointer from the old metadata file to the new one. Everything else in the format, including snapshot isolation, time travel and safe concurrent writers, depends on that single atomic swap being correct. Our Apache Iceberg production deep dive covers the file layout in detail.

For the first years, the pointer lived wherever was convenient. Hive Metastore (HMS) was the incumbent, because Spark and Hive already depended on it. Others used a JDBC database, a DynamoDB table, the AWS Glue Data Catalog, or a directory convention on a file system. Each of these was wired into engines through a Java class in the Iceberg library, so adding an engine meant shipping that class, its dependencies and its credentials into the engine’s runtime. A Rust or Python reader needed its own re-implementation of every catalog.

The Iceberg REST catalog specification replaced that matrix with one HTTP contract documented in the Apache Iceberg REST catalog spec. An engine needs only a URL, an authentication method and a warehouse name. The server behind the URL can be anything that honours the contract. That is why 2024 and 2025 produced a wave of catalog projects, and why the question in 2026 is no longer whether to use a REST catalog but which server to put behind the URL.

Two further pressures raised the stakes. First, the Iceberg table spec itself kept moving: our notes on the Iceberg v3 features and the v4 proposal for single-file commits show that new features land only when both the engine and the catalog understand them. Second, governance moved from the warehouse to the catalog, because a catalog is the only component that sees every engine. If you are still weighing formats, start with the Iceberg vs Delta vs Hudi decision record and return here once Iceberg is the answer.

A note on evidence. Feature claims below come from each project’s own documentation or announcements, linked in Further Reading. Where a capability varies by deployment or version, the text says so rather than asserting a universal. Latency and cost figures are illustrative, labelled as such, and not benchmarks.

What an Iceberg Catalog Actually Does

An Iceberg catalog is a service that maps table identifiers such as sensors.plant1.vibration to the location of the current metadata file, performs the atomic commit that replaces that location, and increasingly handles authorization and storage credential delivery for the engines that call it.

Think of the responsibilities as five layers, and keep them separate when comparing products because vendors blur them.

Figure 1: The Iceberg catalog as control plane. Engines speak one REST protocol; the server owns the commit pointer, policy and credential issuance, and engines read data directly from object storage.

In Figure 1, every engine calls the same REST interface. The catalog service sits between that interface and three back-end concerns: the pointer store, the policy decision, and a credential issuer that talks to the cloud’s token service. Data bytes never flow through the catalog; engines read manifests and Parquet files straight from storage using credentials the catalog handed out.

Layer one is naming and discovery. Namespaces (databases or schemas, possibly nested) and tables are listed, created, renamed and dropped. Most servers also expose table properties, which engines use for write settings such as target file size.

Layer two is the atomic commit. The catalog must provide compare-and-swap semantics on the pointer. A writer prepares new metadata, sends a commit request containing requirements (“the table’s current snapshot must still be X”) and updates (“add this snapshot, set this as main”). The server validates the requirements against the current state and either applies the update or returns a conflict. How that atomicity is achieved, with a transactional database row, a conditional write to object storage or a distributed log, is the main internal difference between products and the main source of operational risk.

Layer three is authorization. Who may list a namespace, read a table, create a table or commit to it? In the legacy model, the answer lived in the storage layer: bucket policies and IAM roles. That works until different teams need different columns or rows of the same table. In the REST model the catalog can evaluate a policy per request, then translate the decision into storage credentials limited to the table’s prefix.

Layer four is credential vending. Instead of every engine holding long-lived storage keys, the catalog issues short-lived, scoped credentials in the loadTable response. This is the feature that turns a catalog from a name server into a security boundary, and a later section examines it closely.

Layer five is optional services. Views, scan planning, change notifications, metrics reports, federation to other catalogs, and Git-like versioning are extras. Support is uneven, which is why capability matrices matter more than logos.

What the catalog does not do

The catalog does not execute queries, compact files, expire snapshots or delete orphan files. Those maintenance tasks run in engines or dedicated services, and some products bundle a scheduler for them while others leave it entirely to you. It also does not make your data fast; planning cost depends on manifest layout, not on which server answered loadTable. Teams sometimes expect a catalog migration to improve query latency. It will not, except through second-order effects such as better caching of table metadata.

The Iceberg REST Catalog Protocol

The REST protocol is a set of HTTP endpoints under a versioned prefix, defined by an OpenAPI document that lives in the Iceberg repository. A client first calls the configuration endpoint, then namespace and table endpoints, then the commit endpoint. Because the spec is published as OpenAPI, conformance can be tested mechanically, and independent compatibility reports exist.

The sequence below shows the read path, which is where authorization and credential vending happen.

Figure 2: The loadTable read path. The catalog authorizes the principal, asks the cloud for a narrowly scoped token, and returns metadata plus credentials in one response.

The configuration handshake

The client calls GET /v1/config, optionally passing a warehouse parameter. The server answers with defaults, overrides and, in current spec versions, an endpoints list naming which operations it supports. The prefix property lets one server host many logical catalogs. That endpoint list matters: a client can see that a server does not offer views or transactions and fall back gracefully, instead of discovering the gap through a 404 in the middle of a job.

Namespaces, tables and views

Namespace endpoints list, create, load, update and drop namespaces, and a namespace can be multi-level. Table endpoints create, load, register, rename, check existence and drop tables. Views follow the same pattern, and a view stores a SQL representation with a dialect, so one view can carry several dialects in principle. In practice, cross-engine views remain the weakest area of interoperability, because each engine parses SQL differently and many servers do not implement the view endpoints at all.

Commit and multi-table transactions

A single-table commit is POST to the table path with a list of requirements and updates. A multi-table commit is POST /v1/{prefix}/transactions/commit with several table changes that must succeed or fail together. This endpoint is the protocol-level basis for atomic cross-table updates, such as writing a fact table and its aggregate in one visible step. Support is not universal, and a server that declares the endpoint still has to implement a transaction across its pointer store. We return to this in the Gotchas section.

Credential vending and remote signing

A client sends the X-Iceberg-Access-Delegation header with vended-credentials, remote-signing or both. With vended credentials, the response config carries short-lived storage credentials such as an AWS session token scoped to the table’s location. With remote signing, the engine never holds credentials at all: it sends each storage request to the catalog’s signing endpoint, which returns a signed request. Vended credentials are simpler and faster; remote signing gives a per-request audit point at the cost of an extra round trip on every file access.

Authentication

The spec supports OAuth2 bearer tokens and the client credentials flow, plus request signing with AWS SigV4 for services that sit behind IAM. The practical consequence: Polaris, Lakekeeper and Unity expect OAuth2-style tokens from an identity provider, while Glue expects SigV4 with the signing name glue, as AWS documents in its Glue Iceberg REST endpoint guide.

Scan planning and newer additions

Recent spec work lets the server plan a scan, returning the file tasks for a query instead of the engine reading manifests itself. This helps thin clients and enables server-side filtering for row and column policies, because the server can refuse to hand out file lists the caller may not see. As of this writing, support differs significantly across servers, so treat it as a capability to test, not assume. A third-party 2026 REST compatibility report found endpoints that some vendors declared but did not serve; read it as a signal to run your own conformance tests, not as a ranking.

The Contenders: What Each Catalog Is in October 2026

Apache Polaris

Snowflake open-sourced Polaris in mid-2024, and it entered the Apache Incubator. On 19 February 2026 the project announced its graduation to a top-level Apache project. For a buyer that matters more than any feature: top-level status means governance by a community PMC rather than by one vendor, which was the main reservation about Polaris in earlier comparisons. The docs list releases through the 1.x line, with 1.4 adding material described below.

Polaris is a full implementation of the Iceberg REST catalog. Its model has principals, principal roles, catalog roles and privileges. A principal role is granted catalog roles, and catalog roles carry privileges on catalogs, namespaces and tables. It supports internal catalogs it manages and external catalogs that federate to another system. According to the Snowflake write-up of 1.4, recent work includes storage-scoped AWS credentials, STS session tags for realm, catalog and table, principal names embedded in sub-scoped credentials to aid auditing, KMS properties in storage configuration, finer SAS token down-scoping for Azure Data Lake Storage, a Helm chart JSON schema, Kubernetes Gateway API support, Postgres optimizations, CockroachDB as a backend and MongoDB as a beta backend. It can also persist Iceberg scan and commit reports to its database in beta.

The implication for operators is that Polaris is a relational-database-backed service. The commit pointer lives in a transactional store, so you inherit the availability and backup characteristics of that database. That is a feature when your platform team already runs Postgres well and a burden when it does not.

Project Nessie

Nessie, from the Dremio team, takes a different premise: treat the data lake like a Git repository. A commit in Nessie records the state of one or more tables; branches and tags are cheap references to commits; merges bring changes from one branch into another. The Nessie documentation describes it as a cloud-native open-source service for Iceberg with Git-inspired data version control and cross-table transactions, and has guides for running it through the Iceberg REST interface. The versioned docs seen at the time of writing run up to the 0.109.x line.

The unique capability is that a branch can span many tables. You can run a nightly transformation on a staging branch that touches 40 tables, validate the result, and merge to main so consumers see all 40 changes at one instant. That is a stronger guarantee than anything a per-table snapshot can give and is the closest the lakehouse gets to a database transaction over a whole schema.

Be careful with status claims here. Nessie’s pace and its position relative to Polaris, which Dremio also supports, are worth checking against the project’s current release notes before committing. The honest summary is that Nessie remains the specialist choice for branch-and-merge workflows, not the default choice for general governance.

Unity Catalog (open source)

Databricks open-sourced Unity Catalog in 2024, and the project is hosted by the LF AI and Data Foundation, part of the Linux Foundation, under Apache 2.0. The project documentation describes a catalog that is compatible with the Hive Metastore API and Iceberg’s REST catalog API and that covers not only tables but unstructured data volumes, functions and AI models. The project describes itself as a sandbox-stage LF AI and Data project, which is a lower maturity tier than a top-level Apache project.

Two different things share the name, and conflating them causes bad decisions. The open-source server is a modest, vendor-neutral catalog. The managed Unity Catalog inside Databricks is a much larger governance platform with lineage, attribute-based access control, system tables and a Databricks-hosted Iceberg REST endpoint. If your estate is Databricks-first, the managed service is the realistic option and the open-source one is a portability hedge. If your estate is not Databricks, evaluate the open-source server on its own merits, which today are narrower than the managed product.

AWS Glue Data Catalog

Glue is the incumbent on AWS and now exposes an Iceberg REST endpoint at https://glue.<region>.amazonaws.com/iceberg, authenticated with SigV4. AWS documents support for the table spec v1 and v2, with v2 as default, for tables in standard S3 and in S3 Tables storage. The documented operations cover configuration, namespace create, list, load, update and drop, and table list, create, load, exists, update and drop. Notably the doc lists only single-level namespaces, so deep namespace hierarchies do not map cleanly.

Glue’s strength is integration: Lake Formation permissions, Athena, EMR, Redshift and many managed engines already speak to it, and there is no server to run. Its weakness is that it is AWS-shaped. Its feature set tracks what AWS ships, not the spec. If the documented operation list above lacks an endpoint you need, such as views or multi-table transactions, you cannot patch the server.

Hive Metastore

HMS still holds an enormous share of production tables. It works with Iceberg through a lock-based commit, and it has no REST interface of its own, no credential vending and a coarse permission model. Keep it where it works, but treat it as the migration source in most plans, not the target. Its one structural advantage is that every Hadoop-era engine can read it.

Lakekeeper

Lakekeeper is an Apache-licensed REST catalog written in Rust on top of iceberg-rust. Its documentation highlights fine-grained permissions inherited down to tables and views, modelled with OpenFGA, with RBAC, relationship-based and attribute-based styles possible, plus an OPA bridge so engines such as Trino can enforce decisions. It supports S3, Google Cloud Storage and Azure storage, and it covers generic tables and unstructured files under the same access control. A commercial tier, Lakekeeper+, adds enterprise features such as Cedar policies. It is the lightest self-hosted option and the one most natural for teams that want a small binary and a strong authorization model.

Other options worth a mention

Gravitino (Apache), Dremio’s built-in Open Catalog and cloud-vendor offerings such as Snowflake’s managed Polaris-based service exist in this space. Several vendors now ship a managed form of Polaris. We do not cover them separately because the architectural questions are the same: what is the commit store, what is the permission model, and who holds the storage credentials.

Side-by-Side Comparison

The table summarizes documented positioning. Cells marked “verify” reflect features that vary by version or deployment; test them in your own environment.

Dimension	Apache Polaris	Nessie	Unity Catalog OSS	AWS Glue REST	Lakekeeper	Hive Metastore
Governance home	Apache top-level project	Open source, Dremio-led	LF AI and Data sandbox	AWS service	Apache-licensed, Vakamo-led	Apache Hive
Iceberg REST	Yes, reference-style	Yes	Yes	Yes, subset	Yes	No
Credential vending	Yes, multi-cloud	Verify	Verify	Via AWS integration, verify	Yes	No
Git-style branches	No	Yes, cross-table	No	No	No	No
Multi-table commit	Verify	Yes, via commits	Verify	Not in documented list	Verify	No
Permission model	Principal roles and catalog roles	Verify	Verify	IAM and Lake Formation	OpenFGA, OPA bridge	Coarse
Federation to other catalogs	Yes	Limited	Limited	Via AWS constructs	Limited	No
You operate a server	Yes, or managed	Yes	Yes	No	Yes	Yes
Multi-level namespaces	Yes	Yes	Verify	Single level documented	Yes	Two levels

The pattern: Polaris and Lakekeeper compete as neutral governance servers, Nessie owns versioning, Unity owns the Databricks gravity well and Glue owns convenience on AWS. No single column dominates, which is why the decision section ends with a flow rather than a winner.

Deep Dive: Credential Vending, Commits and Branches

Credential vending in practice

Without vending, every engine in your estate carries an IAM role or storage key with access to the whole bucket. Revoking one team’s access means editing bucket policy, and a leaked Trino config exposes everything. With vending, the engine authenticates to the catalog only. The catalog authorizes the request, then asks the cloud for a credential limited to the table’s location prefix and valid for a short window.

On AWS the mechanism is the Security Token Service. The catalog assumes a role with an inline session policy that narrows access to the table’s S3 prefix, and the returned token inherits the intersection of the role and that session policy. On Azure it is a SAS token down-scoped to a directory, and on Google Cloud a downscoped token. The Polaris 1.4 work on session tags and principal names in credentials matters for audit: CloudTrail can then show which principal, catalog and table a given storage call belonged to, rather than only a shared catalog role.

Worked example, with illustrative numbers. Suppose 200 Spark executors each open 50 data files per task wave, and a vended token lasts 60 minutes. The catalog issues one token per loadTable call, not per file, so a job that loads 20 tables makes 20 STS calls. That is trivial load. The failure mode is the long job: a streaming writer that outlives its token must call loadTable again to refresh, and engines differ in how well they do that. A Flink job that caches its first credentials for the life of the process will fail on the hour, with a storage AccessDenied that looks nothing like a catalog problem. Test token expiry deliberately, with a short-lived token in staging, before you promise a streaming SLA.

Remote signing removes expiry but adds a hop per request. If a scan touches 5,000 files and the signing call takes a few milliseconds each, the added wall time is non-trivial unless the client batches or parallelizes; this is an illustrative reason many teams prefer vended credentials for batch and reserve remote signing for high-assurance, low-volume access.

Commits, conflicts and the pointer

Every catalog implements the same contract with a different commit store, and that store determines your failure behaviour.

Figure 3: Optimistic concurrency in an Iceberg catalog commit. The writer states a requirement about the base state; the commit store performs a compare-and-swap and returns a conflict when the base has moved.

Figure 3 shows the optimistic loop. Writers never take a lock for the duration of a job. They write new data and manifests, then attempt one atomic swap. If another writer committed first, the swap fails with a conflict (HTTP 409 in the REST protocol), and the writer refreshes the table metadata, checks whether its changes still apply cleanly, and retries. Appends to different partitions usually rebase without trouble. Two jobs that rewrite the same files, such as concurrent compactions or overlapping MERGE statements, cannot both win, and the loser fails after wasted work.

A relational-backed catalog like Polaris performs the swap in a database transaction, so its throughput ceiling is the database’s. A catalog that stores its state in a version-control structure such as Nessie’s commit log performs the swap by advancing a branch head. Hive Metastore uses a metastore lock that is held across the commit, which is why very high commit rates against HMS degrade sharply. In every case, the rule is the same: commit rate per table is bounded, so a streaming writer that commits every second creates 3,600 snapshots an hour on one table. Illustrative arithmetic, but real in shape: that load stresses the pointer store, bloats metadata and demands aggressive snapshot expiry. Most teams settle on commit intervals of tens of seconds to minutes for that reason. The Flink-based streaming patterns we compared elsewhere face exactly this trade-off.

Multi-table commits

A multi-table commit lets a job make changes to several tables visible atomically. The use case is real: a fact table and a dimension updated by one pipeline, or a table and its audit-log twin. The REST spec defines the endpoint, but a server needs a commit store that can apply all-or-nothing across rows or references. Nessie’s model makes this natural because a commit is a set of table states. Relational-backed servers can do it in one database transaction, but only if the implementers chose to. The compatibility report cited earlier marks multi-table transactions as partial even in the reference implementation, so confirm support with a test that deliberately fails the second table and checks that the first did not move.

Branching, tags and the real use of versioning

Branching sounds like a developer convenience; its real value is operational. Three patterns recur.

Write-audit-publish. A pipeline writes to a branch, runs data quality checks against that branch, and merges only on success. Consumers on main never see a half-validated load. Without catalog-level branches, teams emulate this with staging tables and rename swaps, which are not atomic across many tables.

Reproducible training sets. A tag names the exact state of 30 feature tables at the time a model was trained. Per-table time travel can reconstruct one table, but only a catalog-level tag reconstructs all of them consistently.

Safe experimentation. A data scientist works on a branch holding zero copies of data; only new files cost storage, since branches reference existing immutable files.

Iceberg itself has per-table branches and tags inside the table metadata, and engines can use them. Those are scoped to a single table. The cross-table version is what a catalog like Nessie adds, and it is the only reason to choose Nessie over a governance-focused server. If you do not have a write-audit-publish or reproducibility requirement spanning many tables, per-table branches plus good orchestration are usually enough.

Governance, from roles to row filters

Catalog-level governance has three levels of depth.

Coarse object access. A principal can read, write or manage a namespace or table. Polaris and Lakekeeper offer this natively; Glue gets it through IAM and Lake Formation.
Storage enforcement. Vended credentials enforce the same boundary at the object store, so a client that bypasses the catalog still cannot read the files. This is the difference between a policy and a control.
Row and column controls. These require the engine or a server-side planner to apply filters. The catalog can only hand out a file list; it cannot hide a column inside a Parquet file by itself. Row and column policy therefore depends on engine cooperation, on scan planning, or on a proxy layer, and is the least portable level. Lakekeeper’s OPA bridge exists for this reason: engines such as Trino ask the policy engine at query time.

The practical advice is to be honest about which level you need. Most industrial and IoT lakehouses need level two: team-scoped table access with storage enforcement. Few need row-level security on sensor tables. Regulated finance and health data often do, and that tilts the decision toward the platform where the engine-side enforcement already exists.

Choosing: A Decision Flow and Scenarios

The flow below encodes the conclusions of this article in order of how strongly each requirement constrains the choice.

Figure 4: Decision flow. Cross-table branching decides first because only one option offers it; cloud and platform gravity decide next; neutrality and operations decide the rest.

Scenario A: all-AWS industrial lakehouse

Sensor data lands from Kafka into Iceberg on S3, queried by Athena and Spark on EMR, with a handful of dashboard teams. The platform team is small. Glue’s REST endpoint avoids running a server, Lake Formation provides the permission layer, and engines on AWS already know it. The costs: you accept AWS’s feature pace and single-level namespaces. This is the right default, and our Kafka-to-lakehouse change data capture note fits it cleanly.

Scenario B: multi-cloud with Trino, Spark and Snowflake

Three engines, two clouds, and a security team that wants one set of grants. Here a neutral server is the point. Polaris, now an Apache top-level project, fits well, with federation useful when some tables remain in another catalog during the transition. Lakekeeper is the leaner alternative if you prefer a Rust binary and OpenFGA authorization and are comfortable with a younger ecosystem. See our Trino vs Presto vs Spark comparison for the engine side.

Scenario C: Databricks-first enterprise

Most compute is Databricks, and some partners read through other engines. Use managed Unity Catalog for governance and expose Iceberg through its REST endpoint where external engines need access. The open-source server is useful as an exit hedge, but do not mistake it for the managed product’s equal in features.

Scenario D: data-quality-critical pipelines with many dependent tables

A pipeline updates 25 related tables and downstream consumers must never see partial results. This is Nessie’s home ground. Accept that you run a specialist service and that governance features will be thinner than in Polaris, and compensate with storage-level policies.

Scenario E: DuckDB and metadata-in-a-database alternatives

A different architecture stores catalog metadata in a SQL database rather than behind REST. We covered that approach in the DuckLake vs Iceberg catalog analysis. It is a legitimate alternative for single-team analytics, but it gives up the multi-engine ecosystem the REST protocol created, so it is a different decision, not a drop-in swap.

Migration: Moving Between Catalogs Without Moving Data

Because Iceberg data and metadata live in object storage, changing catalogs means re-pointing, not copying. The mechanism is table registration: the new catalog is told “this table’s current metadata file is at this path.” Iceberg exposes this as a register table call, and Spark exposes it as the register_table procedure. The data files do not move, which makes the operation fast and cheap, and it also creates the main hazard.

The two-writer hazard

Registration copies a pointer. After it, the old and new catalogs each believe they own the table. If a job commits through the old catalog while another commits through the new one, the two pointers diverge, and the table’s history forks silently. Neither catalog can detect this because neither sees the other’s commits. Recovery means picking one lineage and discarding the other’s snapshots, which is data loss for whichever jobs wrote to the loser.

The safe sequence is therefore:

Inventory every table and every writer. List which jobs commit to which tables, including maintenance jobs that people forget: compaction, snapshot expiry, orphan-file cleanup.
Freeze writers for one table or one namespace, for a short window.
Register the table in the new catalog, pointing at the latest metadata file.
Verify by reading row counts and the snapshot ID from the new catalog and comparing them to the old one.
Switch writers and readers to the new catalog configuration.
Remove or lock the old entry, so nothing can commit through it again. Revoke its write permission, not just its name.

Migrating from Hive Metastore

HMS to a REST catalog is the most common move. The tooling ranges from the Iceberg register_table procedure to catalog-migration utilities in the Iceberg ecosystem that copy table pointers in bulk. Because HMS permissions are coarse, the larger task is not the pointer move but rebuilding your permission model in the target: map HMS or Ranger policies to Polaris roles or Lakekeeper relations, and test that no principal gained access it did not have before.

Migrating to Nessie or from it

Moving into Nessie is the same register-based flow. Moving out is harder if you used branches: branches and tags exist only in Nessie’s reference model, so you must decide which branch state becomes the target’s truth, and you lose the cross-table history. Export the information you need, such as tags that mark training sets, as table properties or documentation before leaving.

Dual-catalog operation and federation

Some platforms avoid a big-bang migration by federating: the new catalog presents the old catalog’s tables as an external catalog, read-only. Polaris supports external catalogs that federate to other systems. Readers move first; writers move table by table. This costs extra hops and an additional failure domain during the transition, but it limits blast radius. Keep the federation period short and instrumented.

Spec version drift

A migration is also the moment to check table spec versions. Iceberg v3 introduced new capabilities, and our v3 upgrade notes explain the feature gates. A catalog that does not understand a table’s format version may refuse commits or, worse, accept an incompatible update. The compatibility report mentioned earlier found a catalog that refused format-version upgrades through its commit endpoint with a documented account-level error. Check the version your catalog supports before upgrading tables.

Trade-offs, Gotchas, and What Goes Wrong

Catalog availability becomes lakehouse availability. In the legacy model, readers could keep scanning if the metastore blipped, because many engines cached. With REST, loadTable and credential issuance are on the hot path. A catalog outage stops new queries and, if tokens expire, running ones. Design for it: run at least two replicas behind a load balancer, put the backing database on a managed high-availability service, and rehearse failover. For Glue, the equivalent is monitoring AWS service health and regional dependence.

Declared is not served. A server can list an endpoint in its config response without implementing it end to end. The compatibility report’s finding of declared-but-unserved endpoints across several vendors is the reason to write a conformance suite: create a table, commit, evolve schema, create a view, run a multi-table commit with a forced failure, expire snapshots, and run it in CI against every catalog upgrade.

Clock and token skew. Vended credentials rely on time. A node with a skewed clock may reject tokens that are valid, producing intermittent AccessDenied that disappears on retry. Treat NTP monitoring as part of catalog operations.

Permission models do not map one to one. Moving from Lake Formation’s resource tags to Polaris roles, or from HMS to OpenFGA relations, is a redesign. The risk is silent over-grant: a role that looks equivalent but is broader. Diff effective permissions per principal before and after.

Metadata bloat. Frequent commits create many snapshots and manifests. The catalog is not at fault, but it is where the pain surfaces, as slow loadTable responses. Schedule snapshot expiry and manifest rewrites; the cost of ignoring them grows linearly with commit rate.

Federation leakage. When a catalog federates to another, authorization is evaluated twice, in two models. A table may be visible through the federating catalog to a principal that the origin catalog would have refused, if you map service identities too broadly. Use a dedicated, least-privilege identity for federation.

Lock-in is moved, not removed. The REST protocol removes client-library lock-in. It does not remove the lock-in of your policy definitions, your audit history, or vendor-specific extras such as Nessie branches or Unity’s non-table assets. Keep policy as code, export it regularly, and treat vendor extras as deliberate commitments.

Do not assume a managed offering equals the open-source project. Managed Polaris-based services and managed Unity Catalog include extra features, caveats and limits. Read the vendor’s own documentation, not the upstream project’s, for the service you buy.

Practical Recommendations

Start from constraints, not products. Write down your engines, clouds, regulatory needs, team size and whether any workload needs a cross-table atomic publish. Then apply the order from Figure 4. In my experience reviewing lakehouse designs, the most common error is choosing a catalog for its dashboard or for a vendor relationship and discovering later that the permission model cannot express the team structure.

For most new multi-engine platforms outside Databricks, Apache Polaris is the safe default: Apache governance, broad engine support and credential vending across clouds. For an AWS-only platform with a small team, Glue is hard to beat on operational simplicity. Pick Nessie only when you will actually use cross-table branches, and Lakekeeper when you want a lightweight self-hosted server with a strong relationship-based authorization model. Keep Hive Metastore only for tables you have not yet migrated.

Whatever you choose, behave as if you will migrate again: pin your engines to the REST protocol rather than to a vendor-specific catalog class, keep warehouse and namespace naming portable, avoid vendor extension properties on tables where possible, and store policy definitions in version control.

Checklist

[ ] Engine and cloud inventory written down, with the writer list per table.
[ ] Decision flow applied; exceptions documented.
[ ] Conformance suite covering create, commit, schema change, views, multi-table commit, snapshot expiry.
[ ] Credential vending tested with deliberately short token lifetimes, including a long-running streaming job.
[ ] Catalog deployed with at least two replicas and a highly available commit store.
[ ] Backup and restore of the commit store tested, including point-in-time recovery.
[ ] Permission model mapped and effective permissions diffed before cutover.
[ ] Maintenance jobs (compaction, expiry, orphan cleanup) assigned an owner and a schedule.
[ ] Migration plan freezes writers and revokes write access on the old catalog entry.
[ ] Monitoring for commit conflict rate, loadTable latency, token issuance errors and clock skew.

Frequently Asked Questions

What is an Iceberg catalog and why do I need one?

An Iceberg catalog maps table names to the location of each table’s current metadata file and performs the atomic commit that moves that pointer. Without it, concurrent writers could corrupt a table or readers could see inconsistent state. Modern catalogs also enforce permissions and issue short-lived storage credentials, so they act as the security and coordination layer for every engine that touches the data.

Is Apache Polaris production ready after graduating?

Graduation to a top-level Apache project on 19 February 2026 signals community governance and a sustainable contributor base, not a guarantee for your workload. Polaris is widely integrated with Spark, Trino, Flink and Snowflake. Before production, run your own conformance and failover tests, size the backing database for your commit rate, and confirm which features, such as multi-table commits, your version actually serves.

Should I choose Nessie or Polaris?

Choose Nessie if you need cross-table branches and merges, for example write-audit-publish across dozens of tables or reproducible multi-table training snapshots. Choose Polaris if you mainly need a neutral, governed, multi-engine catalog with role-based access and credential vending. If you need both, evaluate the combination against your actual workflows, since the two projects take different approaches and overlap only partly.

Does Unity Catalog open source replace Databricks Unity Catalog?

No. The open-source server is a vendor-neutral catalog supporting Hive Metastore and Iceberg REST APIs and covering tables, volumes, functions and models. The managed Databricks product is a much larger governance platform. Treat the open-source project as a portability option and verify feature parity for anything you rely on, such as fine-grained access control or lineage, before assuming it carries over.

Can I use AWS Glue as an Iceberg REST catalog?

Yes. Glue exposes a REST endpoint at https://glue.REGION.amazonaws.com/iceberg, authenticated with SigV4 using the signing name glue, and supports table spec v1 and v2 with v2 as default. AWS documents namespace and table operations, but only single-level namespaces. Check the documented operation list for features you need, such as views, before relying on it for multi-engine access.

How do I migrate between Iceberg catalogs without copying data?

Register each table in the new catalog by pointing it at the latest metadata file, since the data and metadata stay in object storage. The danger is two catalogs both accepting writes, which forks the table’s history. Freeze writers, register, verify snapshot IDs and row counts, switch clients, then revoke write access to the old entry so nothing commits through it again.