返回

Is your Iceberg catalog healthy?

RSS Snowflake Engineering (Medium) 原文 · 未翻译 发布 2026-11-11 06:38

有效阅读仅保存在当前浏览器,用于「为你」推荐;暂停或清除记录。

来源摘要

A comparison of five open-source Iceberg catalogs on functionality and OSS project health. Apache Iceberg™ allows an organisation to own their data layer in an open format in their own storage, and has become the de facto open table format. Still, every engine must reach those Iceberg tables through

本文目录 7 节
  1. Project health in one image
  2. Who writes the code?
  3. Open source as an exit path
  4. In summary
  5. Methodology
  6. How to reproduce
  7. Sources

原文

A comparison of five open-source Iceberg catalogs on functionality and OSS project health.

Apache Iceberg™ allows an organisation to own their data layer in an open format in their own storage, and has become the de facto open table format. Still, every engine must reach those Iceberg tables through an Iceberg catalog.

A natural extension is an open-source Apache Iceberg catalog to complete the open lakehouse stack, acting as an open governance layer, so that access to your tables isn’t dependent on any one vendor’s managed service.

In this blog, I aim to simplify the decision process when it comes to choosing an Iceberg catalog and therefore building an open lakehouse on Iceberg.

I focus on the functional completeness of each catalog for Iceberg data and dive deep on OSS project health. After all, an OSS project with one vendor driving development carries similar lock-in risk to a proprietary product. My goal here is to provide transparency on the relative health of common OSS catalogs, specifically Apache Polaris™, Apache Gravitino™, Lakekeeper, Unity Catalog OSS and Project Nessie.

The heart of the blog is a collation of public information from each project (which you can review in the public Git repo), but the criteria and analysis are informed by my experience as a specialist architect working with 50+ organisations implementing Apache Iceberg over the past couple of years.

This is a companion post to Iceberg Catalogs: Below the Surface (updated November 2026), which scores six vendor-hosted catalogs and shares the same underlying repo. I split the two posts since the first one was becoming a dissertation!

Disclosure: This is my personal opinion, not representative of the views of my employer. I’m a specialist architect at Snowflake, one of the vendors contributing towards the Apache Polaris project. Every claim below is sourced and reproducible. Don’t take my word for it: test the claims, read the linked project docs, and please comment if you spot any inconsistencies.

Project health in one image

After much iteration, this is the best single view of project health I could come up with: commit velocity against the number of contributing organisations.

If you’re skimming, this is the tl;dr version.

Here, we see Apache Gravitino and Apache Polaris pull ahead of the other assessed catalogs. Both are shipping at ~4x the velocity and with >7x the number of contributing organisations compared to the two relatively inactive projects, Unity OSS and Nessie.

Between Gravitino and Polaris, a subtle difference is reflected in single-vendor concentration: only 15% of all commits for Gravitino came from organisations other than Datastrato, its lead contributor, while Polaris sees 32% of all commits from organisations other than Dremio.

Lakekeeper sits in the middle ground, with 8 contributing organisations and >50% higher velocity than Unity OSS, while still clocking in below 50% of the leading catalogs on both metrics.

For a deep dive on the methodology and the code to reproduce it yourself, please take a look at the underlying repo.

In order to directly compare the catalogs, I also created a tabular version of the project health data, split into 5 criteria for a total of 10 possible points per catalog. This helps to highlight the differences between the various catalogs across various health criteria that don’t all fit on one chart.

We see Apache Polaris (10/10) and Apache Gravitino (9/10) take the lead again. Both demonstrate not only project velocity and diversity, but strong production references: Fivetran / dbt, Cloudera, and Snowflake in the case of Polaris (outside of lead contributor Dremio / SAP), and Pinterest and Bilibili in the case of Gravitino. The strength of Apache project governance also provides a strong foundation for future roadmap openness.

Lakekeeper (5/10) has a strong IRC surface and two public production references (Fresha and Databaas), but remains single-vendor with no foundation backstop.

Unity Catalog OSS (3/10) has improved on vendor diversity since July, though at 8.9% of commits from outside Databricks it sits under the 10% line, and it still has no public production reference (at least, none I could find).

Nessie (0/10) grades zero on all five, and as such I may exclude it from future analyses (anyone have a recommendation for another catalog to take its place?).

The field is closer on capability. Apache Polaris and Lakekeeper implement the full Iceberg REST spec, including multi-table commits, with authorisation and scoped credential vending. Polaris also federates remote IRC catalogs, though federation is off by default. Apache Gravitino federates through its connectors but currently lacks support for multi-table transactions. Unity Catalog OSS serves native Iceberg tables read and write but has no named roles, and Nessie still labels its IRC support experimental.

Who writes the code?

A single company can spread work across many employees and still own nearly all the commits, so headcount is a poor measure of community. The cleanest visual is the share of commits from different organisations, which shows the shared investment and balance in an open-source project. For projects largely driven by a single vendor, this is expected to influence the roadmap and transparency of how it is built.

Apache Polaris stands out on vendor neutrality, with Apache Gravitino an honourable mention. Unity Catalog OSS shows a visible outside share, and Lakekeeper and Nessie remain single-vendor.

Beyond the lead vendor, we see notably more large enterprises contributing towards each of the Apache Polaris (Zscaler, Snowflake, JPMorgan Chase, Bloomberg, Uber, Cloudera, Google, etc.) and Apache Gravitino (Alibaba, Roku, Uber, Cisco, ByteDance, many more) projects compared to others, providing another signal of enterprise readiness:

Open source as an exit path

An open-source catalog provides a clear exit path from a managed service only if the managed service runs the same code as an open-core code, much like we see with other industry favourites such as dbt (hosted via dbt Labs, or self-hostable).

Of the assessed catalogs, Apache Polaris is integrated as the open core Iceberg catalog in multiple managed services: Snowflake Horizon, Dremio Cloud (now part of SAP), dbt / Fivetran, and Cloudera. As the OSS project evolves, these catalogs stand to benefit, and likewise the OSS project is more likely to receive continued contributions from these organisations given their dependency, leading to a mutual rising tide.

Unity Catalog OSS shares a name, a REST API and an Apache Spark™ connector with Databricks Unity Catalog, but the OSS server lacks production-readiness features such as named-role RBAC or federation, and I found no documented migration path from the managed service to the OSS edition. It is not clear whether the OSS project shares code with the proprietary, hosted product that shares its name.

If you’d like to dig deeper on managed catalogs, my companion blog covers this topic in detail.

In summary

Apache Polaris leads the open-source catalogs on both capability and project health, with Apache Gravitino close behind and Lakekeeper one to watch if you can accept a single vendor. Unity Catalog OSS is improving, but is not yet a production choice or an exit from managed Unity. Nessie is converging into Apache Polaris (by their own admission), so start a new project on Apache Polaris instead.

The release notes and contributor data are all public, and the repo has the links and the scripts to check them.

Methodology

  • Capability: rated from released versions and project documentation as of 9 November 2026, checked against the source code of each release.
  • Project health: public git history (commits, excluding merges and bots) and merged pull requests (GitHub API) from 1 May 2025 to 31 October 2026, for the main repository of each project. A trailing 12-month view is kept in the repo as a sensitivity check.
  • Attribution: a contributor counts towards an organisation through a public employer signal, such as a corporate email domain or the employer on their public GitHub profile, or through a curated identity mapping. Other contributors are left unattributed, so outside shares and each leader’s share of all commits are lower bounds. The curated mappings link individuals to employers, gitignored to respect contributors’ privacy.
  • Privacy: results are published as organisation-level counts only; no individual contributor is named.

How to reproduce

git clone https://gitlab.com/hunt.dan/iceberg-state-of-catalogs
cd iceberg-state-of-catalogs

# Commit history for the five open-source catalogs, 18 months (clones the repos on first run)
python3 scripts/fetch_catalog_health.py --since 2025–05–01 --until 2026–11–01 --out data/oss-health-2026–11.json

# Merged pull requests, public signals only (needs the GitHub CLI: gh auth login)
python3 scripts/fetch_pr_attribution.py --since 2025–05–01 --until 2026–11–01 --extra-window 2025–11–01 2026–11–01 --out data/oss-pr-health-2026–11.json

# Charts, then the health score, capability and other tables (the tables need Google Chrome)
CATALOG_HEALTH_DATA=data/oss-health-2026–11.json CATALOG_HEALTH_OUT=docs/assets/oss-2026–11 python3 scripts/plot_catalog_health.py
python3 scripts/render_oss_tables.py data/oss-health-scorecard-2026–11.yaml docs/assets/oss-2026–11/

All figures land in docs/assets/oss-2026–11/; the per-cell evidence is in data/oss-catalogs-2026–11.yaml and data/oss-health-scorecard-2026–11.yaml. There are bound to be gaps in such a fast-moving and complex domain. If you spot something that’s missing or misrepresented, please drop a comment below. Keen to hear your thoughts!

Sources

Open-source catalogs: Apache Polaris · Unity Catalog OSS · Apache Gravitino · Lakekeeper · Project Nessie.

Production references (Polaris): Fivetran, “Getting Started with Apache Polaris Catalog in Fivetran’s Managed Data Lake Service” · Dremio, “Why Dremio Co-Created Apache Polaris and Where It’s Headed” · Cloudera, “Cloudera Adopts Apache Polaris to Advance Open, Governed Data Access” · SAP, “SAP to Acquire Dremio” — “With Dremio, SAP will deliver a universal, open catalog built on Apache Polaris and the open Apache Iceberg REST Catalog API. It serves as both the discovery and semantic layer of SAP Business Data Cloud”.

Apache®, Apache Iceberg™, Apache Polaris™, Apache Gravitino™ and Apache Spark™ are trademarks of The Apache Software Foundation. Unity Catalog, Lakekeeper and Project Nessie are trademarks of their respective owners. Use here is nominative and does not imply endorsement.

Is your Iceberg catalog healthy? was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

这篇内容对你有用吗?

反馈只用于改善内容筛选,不等同于收藏

分享这条资讯
分享海报
保存图片
iOS 也可以长按图片保存