Apache Iceberg Deep Dive: Building a Production Data Lakehouse from Catalog to Query Engine

Last Updated: October 2026

A data lake on object storage has a quiet design flaw: the only thing that defines a “table” is a directory listing. Two jobs writing at once can leave readers seeing half a commit, a rename breaks every downstream query, and “what did this table look like last Tuesday” has no answer. Apache Iceberg exists to fix exactly that, by turning a pile of Parquet files into a table with a versioned, immutable metadata tree and a single atomic pointer.

It matters more in late 2026 than when most tutorials were written. The format has moved on: version 3 of the specification is closed and shipping in the Java library, deletion vectors replace the older position-delete files, Apache Polaris is a top-level Apache project, and release 1.12.0 landed at the end of September. If your mental model of Iceberg is still “Hive tables with snapshots”, several of your defaults are now wrong.

This guide builds the picture from the bottom up and then runs it in production terms: the metadata tree, the commit protocol, partition and schema evolution, row-level deletes, catalogs, maintenance, and an IoT time-series lakehouse example with runnable code. The thesis: Iceberg is not a storage engine, it is a commit protocol plus a metadata index, and almost every production incident traces back to misunderstanding one of those two things.

What this covers: the three-layer metadata design, optimistic concurrency, schema and partition evolution, v2 versus v3 row-level changes, catalog choices, compaction and snapshot hygiene, comparison with Delta Lake and Hudi, and a worked sensor-telemetry pipeline.

Context and Background

Iceberg started at Netflix around 2017 to solve problems Hive tables could not: slow partition listing on S3, unsafe concurrent writes, and schema changes that required rewrites. It was donated to the Apache Software Foundation and became a top-level project in 2020. The key idea was to track files, not directories. A table is a list of data files with statistics, and the list itself is versioned.

Three table formats now dominate the “lakehouse” conversation: Apache Iceberg, Delta Lake (Databricks, now under the Linux Foundation) and Apache Hudi. All three add transactions and metadata over Parquet on object storage. They differ in how they commit, how they handle updates, and how neutral their ecosystems are. Iceberg’s distinguishing bet has been an engine-neutral specification with a REST catalog protocol, so that Spark, Flink, Trino, Snowflake, BigQuery, DuckDB and others can all treat the same table as first-class. The authoritative description is the Apache Iceberg table specification, and it is worth reading the sections on snapshots and commit conflict resolution at least once.

For industrial and IoT teams the relevance is concrete. Telemetry arrives continuously, schemas drift as firmware changes, and the same data feeds dashboards, anomaly models and compliance audits. A time-series database such as those compared in our time-series database internals guide is excellent for hot, recent queries, but years of history belong in cheap columnar storage with open access. Iceberg is the table layer that makes that history queryable by many engines without copying it.

One scoping note. Iceberg is a table format, not a query engine and not a catalog product. It defines how metadata and files are laid out and how a commit works. Engines and catalogs are separate, replaceable choices, and keeping that separation clear is what makes the architecture portable.

The Core Architecture: A Metadata Tree With One Mutable Pointer

Direct answer: an Iceberg table is a catalog pointer to a metadata JSON file, which references snapshots, each snapshot references a manifest list, manifest lists reference manifests, and manifests reference data and delete files with per-file statistics. Every layer below the catalog is immutable; a commit writes new files and then atomically swaps the pointer.

Figure 1: The Apache Iceberg metadata tree. Only the catalog pointer is mutable; everything beneath it is written once and never modified.

The diagram shows the lookup path a query engine walks. It asks the catalog for the table’s current metadata location, reads the metadata file, picks a snapshot, reads that snapshot’s manifest list, filters manifests, and only then opens the data files that survive pruning. The long-description version: five hops from a table name to a Parquet file, four of them reading small immutable files that can be cached aggressively.

The catalog: one pointer per table

The catalog stores, for each table identifier, the location of the current metadata file and nothing else of substance. That is deliberate. Object stores generally lack atomic multi-object transactions, but a catalog (a database row, a REST service, a Glue entry) can do an atomic compare-and-swap on one value. Iceberg reduces the entire commit problem to that single operation: “set the pointer to the new metadata file, but only if it still points to the old one.”

This is why the catalog is the true consistency boundary. Lose the data files and you lose data; lose the pointer and you lose the table’s identity, though the metadata files remain on storage and can be re-registered. Choosing a catalog is therefore choosing your concurrency authority, your access-control plane and, often, your vendor gravity.

The metadata file: schema, specs and snapshot log

The metadata file is a JSON document. It holds the table UUID, the format version, the current and historical schemas, the current and historical partition specs, sort orders, table properties, the list of snapshots, a snapshot log, and a pointer to the current snapshot. Because it records all schemas and partition specs ever used, old snapshots remain readable under the schema they were written with.

Two properties follow. First, every commit produces a new metadata file, so the number of metadata files grows with commit frequency, which is why snapshot expiration and metadata cleanup matter for streaming tables. Second, every column carries a unique integer field ID. Readers resolve columns by ID, not by name or position, which is the mechanism behind safe renames and reordering.

Snapshots, manifest lists and manifests

A snapshot is the complete state of the table at a moment: a manifest list plus a summary (operation type, added and removed file counts, record counts). The manifest list is an Avro file listing the manifests that make up the snapshot, with partition-range summaries per manifest so a planner can discard whole manifests without opening them.

A manifest is an Avro file listing data files or delete files. For each file it records the path, format, partition tuple, record count, size, and column-level lower and upper bounds, null counts and value counts. Those statistics are what let an engine skip files for a predicate like device_id = 'pump-17' or ts >= '2026-10-01' without touching the data. Planning cost scales with the number of manifests and files that survive each pruning stage, not with table size in bytes.

Sequence numbers, introduced in format version 2, order commits. Each data and delete file carries the sequence number of the snapshot that added it, which is what lets a delete apply only to data older than the delete itself. We return to this in the section on row-level changes.

Deeper Analysis: Commits, Evolution, Deletes and Catalogs

The commit protocol: optimistic concurrency

Iceberg uses optimistic concurrency control. A writer reads the current metadata, writes its new data files and manifests to storage, builds a new metadata file, and asks the catalog to swap the pointer from the base version to the new one. If another writer committed in between, the swap is rejected.

Figure 2: Two writers race to commit from the same base. The catalog accepts one compare-and-swap; the loser validates and rebases onto the new head.

On rejection, the losing writer does not necessarily fail. It reloads the new head, then runs validation appropriate to its operation. A pure append only needs to confirm nothing it depends on changed, so it can rewrite its manifest list on top of the new snapshot and retry. A MERGE or DELETE must additionally verify that no concurrent commit added or removed files that match its filter, because otherwise it could delete rows based on stale reads. If validation fails, the commit raises a conflict exception and the job decides whether to rerun.

The default isolation is serializable for some operations and snapshot for others, selectable per operation through table properties such as write.delete.isolation-level and write.update.isolation-level. Snapshot isolation allows more concurrency but permits write skew; serializable is stricter and conflicts more. The commit retry count and backoff are also table properties (commit.retry.num-retries, commit.retry.min-wait-ms and related), and under heavy contention they, not the data volume, set your write throughput ceiling.

A subtle point from the 1.10 era is worth remembering. The REST client no longer retries commits after a 5xx response, because after a server error the client cannot know whether the commit applied, and blindly retrying could produce conflicting operations (per the Google open source blog on Iceberg 1.10). The commit state is ambiguous, and the safe response is to reload and check. Treat this “unknown outcome” case as a first-class failure mode in your own orchestration code too.

Schema evolution by field ID

Because data files store field IDs, schema changes are metadata operations. Adding a column assigns a fresh ID; old files simply lack it and read as null (or as a default value in v3, which supports default values). Dropping a column removes it from the current schema while old files keep the bytes until rewritten. Renaming changes the name attached to an ID. Reordering changes only presentation. Widening types such as int to long, or float to double, is allowed; narrowing is not.

The safety property is that an ID is never reused. This is what prevents the classic Hive failure where dropping and re-adding a column by the same name silently resurrects old data. The cost is that you must evolve through Iceberg APIs or SQL (ALTER TABLE), not by rewriting Parquet files behind its back, and that engines must resolve by ID, which all mainstream integrations do.

Hidden partitioning and partition evolution

Hive makes you add a derived dt column and remember to filter on it. Iceberg stores a partition spec of transforms over source columns: year, month, day, hour, bucket(N, col) and truncate(W, col). Writers apply the transform; readers need only filter on the original column, and the planner translates ts >= '2026-10-01' into partition predicates automatically. Nobody can forget the derived column because it does not exist.

Partition evolution follows from the manifest design. A table can change from day(ts) to hour(ts), or add bucket(16, device_id), as a metadata-only change. Old files keep their old spec; new files use the new one. The planner evaluates the predicate once per spec and unions the results. The practical consequence is that you can start coarse and refine as data volume grows, without a rewrite. The caveat is that files under different specs coexist, so compaction is the natural moment to migrate old data to the new layout if scan performance requires it.

Row-level changes: from position deletes to deletion vectors

Parquet files are immutable, so updating or deleting a row means one of two strategies. Copy-on-write rewrites every affected data file without the changed rows: simple reads, expensive writes. Merge-on-read writes a small delete record and reconciles at read time: cheap writes, more work per read until compaction.

Format version 2 introduced merge-on-read through two delete file types. Equality deletes say “remove rows where key = X”, common in streaming upserts, and cost a join against data at read time. Position deletes say “remove row N of file F”, cheaper to apply but numerous: a single commit can emit one position-delete file per data file touched, and over time the delete files themselves become the small-files problem.

Format version 3 replaces position-delete files with deletion vectors. A deletion vector is a compressed bitmap of deleted row positions for one data file, stored in a Puffin file as a blob of type deletion-vector-v1. There is at most one active deletion vector per data file, so a later delete merges into the existing vector instead of piling up another file. Readers apply it by a bitmap lookup rather than a sort-merge of position lists. The 1.10 release notes describe deletion vectors as ready for production use, and the format closes the long-standing weakness of merge-on-read, unbounded delete-file accumulation.

Version 3 also adds row lineage: each row can carry a stable _row_id and a _last_updated_sequence_number, derived from per-file first-row-ID assignments in metadata rather than stored redundantly in every row. That gives change-data-capture consumers and incremental materialized views a reliable answer to “which rows changed since snapshot S” without diffing. Other v3 additions include the variant type for semi-structured data, geometry and geography types, nanosecond-precision timestamps, default column values and table encryption keys.

Upgrading is one-way and explicit. Per the 1.11 coverage by Data Lakehouse Hub, you set format-version to 3 with ALTER TABLE ... SET TBLPROPERTIES, which is metadata-only and leaves existing data files in place; subsequent writes then use v3 features. Upgrading the library does not upgrade tables. Before flipping the switch, confirm that every engine reading the table supports v3, because an older reader will refuse a v3 table rather than misread it.

Catalogs: REST as the integration point

The REST catalog specification turned catalog access into an HTTP API: load table, create table, commit updates, list namespaces, plus credential vending and, in recent versions, view and scan-planning endpoints. Any engine that speaks it can reach any compliant catalog without bundling vendor SDKs.

The main options in 2026 are these. Apache Polaris is the open implementation of the REST spec; according to a July 2026 ecosystem review it graduated to a top-level Apache project in February 2026, was at release 1.6.0 by the end of June, and provides role-based access control, short-lived credential vending, federation to external catalogs (Hive Metastore, AWS Glue, other REST catalogs, BigQuery Metastore), generic tables so Delta and Hudi tables can be registered, and a policy store for maintenance policies. Project Nessie adds Git-like branches and tags, useful for isolated ETL testing. AWS Glue Data Catalog is the zero-ops choice on AWS. Unity Catalog, Apache Gravitino, Lakekeeper and managed offerings such as Snowflake Open Catalog and Dremio’s catalog fill the rest of the map. The Polaris project site is at polaris.apache.org.

Credential vending deserves emphasis because it changes the security model. Instead of giving every engine long-lived storage keys, the catalog hands each request a short-lived credential scoped to one table’s paths. Access control moves from the bucket policy, which is coarse, to the catalog, which knows tables and roles.

Engines: who reads and writes what

Iceberg’s engine story is about contracts. Spark is the workhorse for batch ingestion, maintenance procedures and ML feature jobs. Flink is the streaming writer; the Flink Iceberg sink commits on checkpoint, so the checkpoint interval is also your commit interval and your data-freshness floor. Trino and similar distributed SQL engines serve interactive queries, relying on manifest statistics for pruning. DuckDB and PyIceberg serve laptops, notebooks and small services. Snowflake and BigQuery can read, and increasingly write, Iceberg tables through catalog integrations. See our Apache Flink 2 stateful stream processing guide for the streaming side.

Version compatibility is the part that bites. The 1.12.0 release, which Dremio’s upgrade guide dates to September 30, 2026, removed Spark 3.4 and Flink 2.0 support, leaving Spark 3.5, 4.0 and 4.1 and Flink 1.20, 2.1, 2.2 and 2.3 as the supported lines. The 1.11.0 release in May 2026 had already dropped Java 11 in favour of Java 17 or 21, and deprecated writing position deletes that embed row data. Pin your engine, Iceberg runtime jar and catalog versions together, and test upgrades on a copy of a real table.

Release 1.12 also keeps maturing the new features. Its notes cover Variant read and write paths in Spark and Flink, geometry and geography support, correctness fixes for row lineage, deletion vectors and encryption, Hilbert-curve clustering in the Spark rewrite_data_files procedure, and groundwork for a future format version 4. The same notes state that v4 is not production-ready and that 1.12 does not create v3 tables by default. Treat v4 items as roadmap, not features.

Table maintenance is not optional

Iceberg’s immutability means that every write adds files and nothing is ever removed automatically. A healthy table needs a maintenance loop.

Figure 4: The maintenance loop. Compact small files, fold deletes, expire snapshots, rewrite manifests, and only then remove orphan files.

The order matters. Compaction (rewrite_data_files) merges small files toward a target size, 512 MB by default via write.target-file-size-bytes, and can bin-pack, sort, or Z-order the result. For a streaming table checkpointing every minute, you create at least 1,440 files per partition per day per writer task before any compaction; at 128 writer tasks that is nearly 185,000 files daily, which is precisely the case that makes planning slow. The arithmetic is simple and the remedy is scheduled compaction on closed partitions, which also avoids conflicts with the writers still appending to the current hour.

Snapshot expiration (expire_snapshots) removes old snapshots and the files only they reference, bounding both storage and metadata growth. It defines your time-travel window, so set it from a requirement, such as seven days of rollback plus whatever audit retention you owe, not from habit. Manifest rewriting (rewrite_manifests) regroups manifests so that partition-aligned pruning works well after many small commits. Orphan file removal (remove_orphan_files) deletes files in the table location that no metadata references, left by failed jobs. Run it last and with an age threshold of days, never minutes, because a file written by an in-flight commit looks orphaned until the commit lands. Deleting too eagerly here is the most common way to corrupt a table.

Polaris and some managed catalogs can now host maintenance policies, and cloud services such as Amazon S3 Tables run compaction for you, but the same questions apply: what is the target size, what is the retention, and who owns the schedule.

Time travel, branches and rollback

Because snapshots are immutable and addressable, reading history is a metadata lookup. In Spark SQL, SELECT * FROM t TIMESTAMP AS OF '2026-10-01 00:00:00' or VERSION AS OF <snapshot-id> reads an earlier state; CALL system.rollback_to_snapshot(...) moves the table back by creating a new commit that points to an old snapshot. Branches and tags, part of the spec since v2, let you stage writes on a branch, validate them, and fast-forward, a pattern known as write-audit-publish. This is the single most useful pattern for data quality: ingest to an audit branch, run checks, publish to main only when they pass.

A Worked Example: An IoT Time-Series Lakehouse

Consider a plant with 4,000 machines, each emitting about 20 tags once per second. That is 80,000 readings per second, or roughly 6.9 billion rows per day. The goal: dashboards on the last week, model training on years of history, and an audit trail for quality investigations.

Figure 3: A telemetry lakehouse. Data flows from the Unified Namespace through Kafka and Flink into an Iceberg table governed by a REST catalog and read by multiple engines.

The path in Figure 3 starts at the edge. PLC and sensor values are published into a Unified Namespace on an MQTT broker, as covered in our Unified Namespace architecture guide. A bridge lands them on Kafka topics (the trade-offs between brokers are discussed in our Kafka vs Redpanda vs WarpStream ADR). A Flink job writes to Iceberg, and Trino and Spark read through a REST catalog.

Table design

Partitioning is the decision that matters most. Partition by hours(ts) plus bucket(16, machine_id). The hourly transform lets time-bounded queries prune hard; the bucket spreads each hour’s writes across 16 groups so no single file grows unmanageably and per-machine queries read about one sixteenth of the hour. Avoid partitioning directly by machine_id: with 4,000 values it multiplies partitions and small files by 4,000. Sort within files by machine_id, ts so column statistics are tight and range predicates skip effectively.

At 6.9 billion rows per day, with a few dozen bytes per row compressed in Parquet, expect on the order of a few hundred gigabytes per day. That figure is an illustration and depends on tag types and compression, not a benchmark; measure your own data. With a 512 MB target file size that is on the order of several hundred files per day after compaction, which is a comfortable manifest count.

-- Spark SQL: create the raw telemetry table (format version 3)
CREATE TABLE lake.plant.telemetry (
  ts          TIMESTAMP,
  machine_id  STRING,
  tag         STRING,
  value       DOUBLE,
  quality     INT,
  payload     VARIANT
)
USING iceberg
PARTITIONED BY (hours(ts), bucket(16, machine_id))
TBLPROPERTIES (
  'format-version' = '3',
  'write.target-file-size-bytes' = '536870912',
  'write.parquet.compression-codec' = 'zstd',
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'commit.retry.num-retries' = '10'
);

ALTER TABLE lake.plant.telemetry WRITE ORDERED BY machine_id, ts;

The variant column holds the occasional vendor-specific blob without forcing a schema change, an illustration of v3’s semi-structured support. Confirm that your Spark and Flink versions support the type before using it; 1.12 notes describe vectorized Variant reads in Spark and Variant read/write in Flink.

Streaming ingestion and its trap

The Flink sink commits on checkpoint. With a one-minute checkpoint you get one-minute freshness and about 1,440 commits per day per table. Ten-second checkpoints give better freshness but ten times the commits, metadata files and small files. Pick the slowest interval your freshness requirement tolerates, and let compaction do the rest. If dashboards need second-level data, serve that from the time-series store and let the lakehouse own history, which is the split we recommend.

For late data and corrections, use upserts sparingly. An equality-delete upsert stream is expressive but expensive to read; Flink 1.12 support for converting equality deletes into deletion vectors in compaction is a useful mitigation. For immutable telemetry, append-only is far cheaper and almost always correct: model corrections as new rows with a version column and resolve them in a view.

Querying and maintenance in practice

# PyIceberg: read a REST-catalog table with predicate pushdown
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "plant",
    **{
        "type": "rest",
        "uri": "https://polaris.example.com/api/catalog",
        "credential": "client_id:client_secret",
        "warehouse": "plant_lake",
        "scope": "PRINCIPAL_ROLE:ALL",
    },
)
tbl = catalog.load_table("plant.telemetry")

scan = tbl.scan(
    row_filter="machine_id = 'pump-17' AND ts >= '2026-10-01T00:00:00'",
    selected_fields=("ts", "tag", "value"),
)
df = scan.to_pandas()
print(len(df), "rows;", len(scan.plan_files()), "files planned")

Printing the planned file count is a cheap health check: if a one-machine, one-day query plans thousands of files, your layout or compaction is off. The same check in SQL is a query against the files metadata table.

-- Inspect health through Iceberg metadata tables (Spark/Trino)
SELECT count(*) AS files, avg(file_size_in_bytes)/1048576 AS avg_mb
FROM lake.plant.telemetry.files;

SELECT committed_at, snapshot_id, operation, summary['added-data-files'] AS added
FROM lake.plant.telemetry.snapshots ORDER BY committed_at DESC LIMIT 10;

-- Nightly maintenance (Spark procedures)
CALL lake.system.rewrite_data_files(
  table => 'plant.telemetry',
  where => 'ts < current_timestamp() - INTERVAL 2 HOURS',
  options => map('target-file-size-bytes','536870912')
);
CALL lake.system.expire_snapshots(
  table => 'plant.telemetry',
  older_than => TIMESTAMP '2026-10-02 00:00:00', retain_last => 20
);
CALL lake.system.rewrite_manifests('plant.telemetry');
CALL lake.system.remove_orphan_files(
  table => 'plant.telemetry',
  older_than => TIMESTAMP '2026-10-06 00:00:00'
);

Note the compaction where clause excludes the last two hours, so maintenance does not fight the streaming writer over files still being appended. The dates are examples; compute them from the run time.

Governance and audit

Register the table in the REST catalog and define roles: a plant_reader role with read on the plant namespace for dashboards, a ml_engineer role with read on raw and write on a features namespace, and a data_platform role that owns the namespace and maintenance. Engines receive vended credentials scoped to the table’s storage paths. For quality investigations, the snapshot history is the audit log: given a defective batch of product, time travel to the table as it existed when the batch was processed and reproduce the model input exactly, which is far stronger than “we retrained last month”.

Iceberg vs Delta Lake vs Hudi

All three provide ACID commits on object storage. The differences are in the details that determine operations.

Dimension	Apache Iceberg	Delta Lake	Apache Hudi
Commit mechanism	Atomic catalog pointer swap over immutable metadata tree	Ordered transaction log of JSON actions with periodic checkpoints	Timeline of instants on the table’s metadata directory
Concurrency	Optimistic, per-operation validation	Optimistic, conflict checks on log	Optimistic and non-blocking options
Row-level changes	Copy-on-write, merge-on-read, v3 deletion vectors	Copy-on-write, deletion vectors	Copy-on-write and merge-on-read (log files)
Partitioning	Hidden transforms with in-place evolution	Partition columns, plus liquid clustering	Partition paths with index options
Engine neutrality	Specification and REST catalog designed for many engines	Strongest on Spark and Databricks; broad connectors	Strong on Spark and Flink
Strength	Multi-engine interop, evolution, governance via catalog	Tight Databricks integration	Upserts and incremental pull on streaming workloads

Treat the table as a map of design intent, not a scoreboard. Delta Lake has also invested in deletion vectors and has a compatibility layer (UniForm) that exposes Delta tables with Iceberg metadata, and Polaris’s generic tables let multiple formats share one catalog. The convergence is real, which lowers lock-in risk. Choose on your engine mix and catalog strategy: if the estate is predominantly Databricks, Delta’s integration depth may beat neutrality; if you want Trino, Flink, Spark and a warehouse reading the same tables, Iceberg’s specification is the strongest fit. Features and maturity move quickly, so verify against current documentation before committing.

Trade-offs, Gotchas, and What Goes Wrong

Small files are still the first failure. Frequent commits create many small data files and many metadata files. Planning gets slow, S3 request costs climb, and queries degrade long before storage becomes a concern. The cure is compaction, sensible checkpoint intervals and partition granularity that matches data volume.

Over-partitioning. Partitioning by a high-cardinality column, or by hours on a table that gets a few megabytes an hour, produces more partitions than data. Start coarse; partition evolution lets you refine later.

Orphan cleanup races. Running remove_orphan_files with a short threshold can delete files from an in-flight commit and corrupt the table. Use an age of several days and never run it concurrently with a long writer you do not control.

Over-eager snapshot expiration. Expiring snapshots removes time travel and breaks any branch or tag reference or long-running reader still using them. Align retention with audit and rollback requirements, and keep a minimum number of snapshots.

Commit contention. Many writers on one table fight over the same pointer. Aggregate writes (fewer, larger commits), partition writers so their changes do not conflict, or split into multiple tables. The unknown-outcome case after a server error needs explicit handling: reload, check whether your snapshot is present, and only then retry.

Equality-delete read cost. Upsert-heavy streaming with equality deletes makes every read pay a join. Use deletion vectors where the engine supports them, compact often, and keep primary-key-style tables narrow.

Version skew. Mixing engine versions across v2 and v3 tables is the most likely cause of a surprising outage. A reader that does not understand a table’s format version fails the read. Keep a compatibility matrix and upgrade readers before tables.

Metadata is not a backup. Time travel protects against logical mistakes, not against deleting the bucket. Use object-store versioning or replication, and remember that expired snapshots and removed files are gone.

Catalog lock-in. The data files are open Parquet, but governance lives in the catalog. A REST-spec-compliant catalog eases migration, though roles and policies do not always move with it. Plan the exit path before you rely on proprietary governance features.

Honest limits of the format. Iceberg is an analytical table format. It is not a low-latency serving store, and single-row point lookups or sub-second freshness belong elsewhere. For wide-column operational telemetry stores, see our ScyllaDB vs Cassandra vs DynamoDB ADR.

Practical Recommendations

Start with the catalog, because it is the hardest decision to reverse. Choose a REST-compatible catalog, ideally Apache Polaris or a managed service built on it, so engines, credentials and access control are all uniform. Decide who owns the maintenance schedule before the first production write.

Create new tables on format version 3 only if every engine and tool that will touch them supports it; otherwise stay on version 2 and plan the upgrade as an explicit, tested step. Prefer append-only models and deletion vectors over equality-delete upserts. Partition by a time transform plus, if needed, a bucket, and sort within files.

Treat the table as a product: write a runbook with target file size, checkpoint interval, snapshot retention and orphan-file thresholds, and monitor the files and snapshots metadata tables for file counts, average size and commit rates.

• Pick a REST catalog; use credential vending; define roles per namespace.
• Set write.target-file-size-bytes, compression and sort order explicitly.
• Choose checkpoint intervals from freshness needs, not defaults.
• Schedule compaction, snapshot expiration, manifest rewrite and orphan removal, in that order.
• Use write-audit-publish branches for quality gates.
• Pin and test engine, runtime and catalog versions together before every upgrade.
• Replicate or version the bucket; time travel is not a backup.

Frequently Asked Questions

What is Apache Iceberg used for?

Apache Iceberg is an open table format that gives files on object storage the behaviour of database tables: atomic commits, schema evolution, hidden partitioning, snapshots for time travel, and row-level updates and deletes. It lets Spark, Flink, Trino and other engines query and modify the same data safely. Typical uses are analytics lakehouses, streaming ingestion, ML feature history and audit-grade data retention.

Is Apache Iceberg a database or a file format?

Neither. It is a table format: a specification for metadata that tracks which data files make up a table at each point in time. The data itself is stored in ordinary files, usually Parquet, with Avro and ORC also supported. There is no server to run for the format itself; you need an engine to read and write and a catalog to hold the table pointer.

What is new in Iceberg format version 3?

Version 3, which Iceberg 1.10 treated as closed for implementation, adds deletion vectors, row lineage, the variant type for semi-structured data, geometry and geography types, nanosecond timestamps, default column values and table encryption keys. Upgrading a table is an explicit, one-way metadata change, and every reader of that table must support v3. Upgrading the library alone does not change existing tables.

How do you handle small files in Iceberg?

Reduce them at the source with longer streaming checkpoint intervals and appropriate partition granularity, then fix the rest with scheduled compaction via rewrite_data_files, targeting around 512 MB per file by default. Follow compaction with snapshot expiration, manifest rewriting and orphan file removal. Compact closed partitions so maintenance does not conflict with active writers.

Which catalog should I use with Iceberg?

Pick a catalog that speaks the Iceberg REST specification. Apache Polaris is the open-source option, with role-based access control, credential vending and federation to other catalogs. Project Nessie suits branch-based workflows, AWS Glue suits AWS-native teams, and Unity Catalog or managed services fit specific platforms. Choose by engine mix, governance needs and how easily you can migrate later.

Iceberg or Delta Lake?

Both are production-grade. Iceberg’s strength is a vendor-neutral specification and REST catalog that many engines support, which favours mixed-engine estates. Delta Lake is deepest on Databricks. Compatibility layers such as Delta UniForm and catalog-level generic tables are narrowing the gap, so decide on your engines, catalog and team skills rather than on a feature checklist, and verify against current docs.

Further Reading

• Time-series database internals: InfluxDB, TimescaleDB and QuestDB for the hot-path store that complements an Iceberg history layer.
• Unified Namespace architecture for industrial IoT for the source of the telemetry in the worked example.
• Apache Flink 2 stateful stream processing for the streaming writer.
• Kafka vs Redpanda vs WarpStream for edge telemetry for the transport layer.
• Apache Iceberg table specification and Iceberg documentation, the primary sources for format and procedures.
• Apache Polaris, the open REST catalog project.

By Riju — about