Lakehouse table formats and catalogs
Compare how table state is committed, read and recovered.
Parquet describes files. A table format adds the metadata and rules needed to decide which files represent a table at a particular point in time. The query engine executes work, and a catalog helps locate and manage the table. Those responsibilities explain why Iceberg and DuckLake belong here.
Sources reviewed: 2026-09-06. This is a documentary evaluation, not a benchmark. Recommendations depend on the reader/writer combination and operating scenario. A library release is not a table specification version.
At a glance
Rating (1–5): an editorial readiness score for the fit named in the same row, dated 2026-09-06. It adds five criteria worth 0, 0.5 or 1 point each: maintenance, open edition (what the open-source edition includes without a paid tier), maturity and community, operating scope and interoperability, with caps for archived, stale or prerelease projects. The breakdown sits under the table and the method on the Blog index. A rating is not a benchmark or a universal ranking; the workload sections still decide.
| Project | Rating | Code license | I would shortlist it for | Main constraint |
|---|---|---|---|---|
| Apache Iceberg 1.11.0 | Apache-2.0 | Tables shared by independently chosen analytical engines | Each reader, writer, catalog and format-version combination needs a compatibility check. | |
| Delta Lake 4.4.0 | Apache-2.0 | A platform whose engines and operating practices already support Delta | Enabled table features and reader/writer protocols can narrow compatibility. | |
| Apache Hudi 1.2.0 | Apache-2.0 | CDC, updates and incremental processing over a data lake | Table type, indexing, compaction and cleaning affect freshness and maintenance. | |
| DuckLake specification 1.0 | MIT reference implementation | A lakehouse with table metadata coordinated through a SQL database | Catalog concurrency, recovery and the exact engine integration are central design choices. |
How each rating was computed
Five criteria worth 0, 0.5 or 1 point each. Caps: archived upstream 1, no stable release in 18 months 2, no general-availability release 2.5. "Open edition" scores what the open-source edition includes without a paid tier. Scored on 2026-09-06 from the official repository, releases and documentation; the method is on the Blog index.
| Project | Maintenance | Open edition | Maturity | Operations | Interoperability | Rating |
|---|---|---|---|---|---|---|
| Apache Iceberg | 1 | 1 | 1 | 0.5 | 1 | 4.5 |
| Delta Lake | 1 | 1 | 1 | 0.5 | 0.5 | 4 |
| Apache Hudi | 1 | 1 | 1 | 0.5 | 0.5 | 4 |
| DuckLake | 1 | 1 | 0.5 | 0.5 | 0.5 | 3.5 |
| Apache Polaris | 1 | 1 | 1 | 0.5 | 1 | 4.5 |
| Lakekeeper | 1 | 1 | 0.5 | 0.5 | 1 | 4 |
| Nessie | 1 | 1 | 0.5 | 0.5 | 0.5 | 3.5 |
| Unity Catalog OSS | 1 | 0.5 | 0.5 | 0.5 | 0.5 | 3 |
Where DuckLake fits
DuckLake is an integrated lakehouse table and catalog format. Its specification places table metadata in SQL tables and data in Parquet files. It belongs beside Iceberg, Delta Lake and Hudi, with that architectural difference visible in the comparison.
DuckDB is a query engine; DuckLake is not another name for its native database file. The DuckLake documentation separates the specification from the DuckDB extension. The reference implementation is MIT-licensed. Its version and supported DuckDB build must be checked separately from specification 1.0.
For remote concurrent clients, the catalog guide points to PostgreSQL. DuckDB and SQLite catalogs suit different local deployment patterns. Choosing a SQL catalog does not remove the need to operate and restore that database.
Start with the workload
- Several engines must read and write the same tables: start by testing Iceberg across the exact engine versions. Confirm deletes, schema changes and timestamp semantics, not merely that every product lists an Iceberg connector.
- A Delta-based Spark platform already exists: keep Delta on the shortlist. Test the second engine before enabling table features that its reader or writer cannot handle.
- An update-heavy CDC pipeline: compare Hudi's table and query types against Iceberg or Delta with the same change stream. Include compaction and read freshness in the result.
- A compact lakehouse centered on DuckDB and a shared SQL catalog: trial DuckLake. Check concurrent transactions, metadata recovery and the required integrations before promising cross-engine portability.
Version and operating notes
Iceberg — keep implementation and format versions separate
The specification identifies completed format versions 1, 2 and 3, with version 4 under development at review. That numbering is distinct from the 1.11.0 library release in the table.
Build a reader/writer compatibility matrix for the features actually enabled. Test partition evolution, row deletion and snapshot expiration across engines. A catalog connection succeeding does not establish that every table feature can be read or written correctly.
Delta Lake — table features are a compatibility decision
The table properties reference describes settings including retention and protocol requirements. The transaction protocol defines the requirements readers and writers must follow.
Trial feature upgrades with every consuming engine. Restore transaction history and data files together. Treat retention and vacuum settings as recovery decisions; time travel cannot recover files already removed from all retained copies.
Hudi — compare table type and query semantics
Hudi's table types distinguish Copy on Write and Merge on Read. On Merge on Read, snapshot and read-optimized queries can observe different freshness between compactions.
Replay updates, late events and deletes with the actual record key and ordering rules. Measure ingestion, compaction backlog and queries together. Verify that the consumer's query type satisfies the freshness requirement.
DuckLake — recover the catalog and files as one dataset
The backup guidance covers the catalog database and file storage. Metadata is part of the durable dataset, not a cache that can always be rebuilt by listing Parquet.
Restore into a clean environment, test references to every retained file and reproduce a writer failure around commit. Check the catalog's transaction behavior with the intended number of clients. Include small-write handling and maintenance in the operating trial.
Catalogs are another choice
These projects help manage table discovery or governance. Their license does not automatically describe a vendor's hosted product. Ratings use the same method as the table above.
| Catalog | Rating | Reviewed code license | Role | What I would verify |
|---|---|---|---|---|
| Apache Polaris 1.7.0 | Apache-2.0 | Catalog services for Iceberg | Engine authentication, credential handling and restore of catalog state. | |
| Lakekeeper 0.13.3 | Apache-2.0 | Iceberg REST catalog | Authorization integration, object-store access and metadata recovery. | |
| Nessie 0.108.4 | Apache-2.0 | Versioned catalog with branches and tags | Engine support and the relationship between catalog references and table snapshots. | |
| Unity Catalog OSS 0.6.0 | Apache-2.0 | Open catalog and governance interfaces | Supported APIs, authorization and feature scope of OSS versus the managed service. |
DuckLake's SQL catalog follows its own metadata specification; it is not an interchangeable Iceberg REST catalog. Likewise, Lance is a data format used by LanceDB, Parquet is a file format, and S3 is an object storage API. Keep those layers visible in the technology inventory.
A trial that can change the decision
Use the same mutable event table and two independent clients.
- Commit correctness: interrupt a writer and verify that readers see a coherent committed snapshot, including multi-table requirements if needed.
- Interoperability: read and write with the exact engine combinations; include nulls, timestamps, deletes and schema/partition evolution.
- Freshness and maintenance: measure query visibility alongside compaction, small files, snapshot expiration and metadata growth.
- Authorization and recovery: test catalog access and direct object access, then restore catalog state, credentials and referenced files.
- Exit: export a representative dataset and preserve semantics in the destination. Shared Parquet files do not prove metadata-only migration.
Follow the data
Choose object storage for the files, transformation for their models and an orchestrator for scheduled work. Compare OLAP engines and BI for consumption. A table format does not supply all of those services.