Understanding the Role of Data Catalogs

☠️ Let’s talk about the “catalogs are dead” narrative ☠️ Over the last few years, I’ve seen some data observability companies claim that observability is the future and data catalogs are a thing of the past. I'm still waiting for the catalog industry to dry up ... but it's not. In fact, it's accelerating. Observability tools are awesome — they automate gathering data quality signals and can be a powerful for understanding and monitoring your data. They can overlap with catalogs when it comes to features like data lineage. But here’s the thing: catalogs solve a much broader set of problems. Data catalogs don’t just cater to the data engineer persona—they serve analysts, data stewards, business users, and more. They drive better collaboration, governance, and discovery across the entire organization, helping with discovery, data self-service, compliance, issue resolution, stewardship, data product management, and much more. Observability tools? Their core focus is providing data teams signals on the health and quality within data pipelines. The reality is that catalogs and observability tools aren’t competing. They’re complementary. Observability brings critical monitoring and alerting capabilities to the table. Catalogs bring context, governance, and the glue that connects people to the data they need to make decisions. There are definitely PASSIVE catalogs out there ... they are a dying and commoditizing breed. But ACTIVE catalogs + (ACTIVE) observability is a potent combo and both growing fast. So no, catalogs aren’t dead. They’re evolving to become active metadata and active governance platforms and meet broader use cases that enable entire businesses, not just engineering teams. What do you think—are you Team Catalog, Team Observability, or Team “Why Not Both?” 💬👇

Lakehouse catalogs are the front door to your data. In a modern data lakehouse architecture, catalogs like Polaris, Nessie, Gravitino, Lakekeeper, Unity Catalog, and AWS Glue play a critical role in connecting tools, teams, and data. Here’s what they do: • Discoverability: They provide a searchable inventory of datasets—so users and tools can easily find what exists in your lakehouse. • Metadata Management: They store basic metadata about assets like their location on your data lake so engines can locate the table metadata. • Access Control: They enforce security policies with RBAC (Role-Based), TBAC (Tag-Based), and ABAC (Attribute-Based Access Control)—so only the right users see the right data. • Cross-Tool Insights: They act as a central point of context, making your lakehouse interoperable across analytics engines, catalogs, and governance platforms. • Future-Forward: Expect these catalogs to evolve to support Fine-Grained Access Control (FGAC) rules that engines can interpret natively—bringing column- and row-level security to life. Lakehouse catalogs aren’t just metadata stores—they’re the entry point to your data, enabling safe, governed, and open interoperability across your entire data ecosystem. If the lakehouse is your platform, the catalog is your control tower. #DataLakehouse #ApacheIceberg #DataGovernance #Catalogs #ApachePolaris #ProjectNessie #Gravitino #Lakekeeper #UnityCatalog #AWSGlue #MetadataManagement #RBAC #ABAC #ModernDataStack #Dremio #OpenStandards

Catalogs have a two-way relationship with AI. They must (1) use it to boost efficiency and (2) govern it to reduce risk. Here's how to evaluate their success. This comes from a new BARC article, "Is Your Data Catalog Ready for the AI Age?" by my smart colleagues Florian Bigelmaier and Timm Grosser. Florian and Timm define three levels of AI readiness -- basic, advanced and leading edge -- each of which require the two-way relationship with AI, as outlined in this diagram. Then they recommend questions to ask vendors for each level, focusing on areas such as data discovery and classification, metadata enrichment and data lineage, and so on. Check out the excerpts below and tell us what you think. What level catalog do you need? And what level do you have? "Enterprise data catalogs need to address AI from two perspectives: - "Automating or optimizing data governance, stewardship tasks and user experience - "Supporting the governance of AI models and applications." The following checklist can help determine whether a tool meets basic, advanced, or leading-edge requirements in certain areas. > Data Discovery & Classification "To make a data catalog a great experience for users, automation along the data discovery and classification process is a must. Basic: "Does the catalog automatically detect and register new data sources with minimal manual setup? "Does it provide a straightforward search and browse interface for end users? Advanced: "Does it use ML-based (machine learning) algorithms to infer data relationships? "Can it detect and classify sensitive or PII data accurately, integrating with privacy compliance requirements (GDPR, HIPAA, etc.)? Leading-edge: "Does it support (semi-)automated domain-specific classification (e.g., finance, healthcare) with relevant taxonomies? > AI Model Governance "The scope of data governance is expanding as AI governance has become an additional requirement. Basic: "Does the catalog provide standard data governance features? (e.g., create and execute workflows, stewardship dashboards, data quality features) Advanced: "Does the catalog offer robust model governance for ML/AI models, including versioning, model profiling with bias detection, and performance monitoring? "Is there a tight link between existing data governance practices and ML/AI model governance, such as through automatic propagation of tags from data to the derived model, alerting the model owner when the data quality of the underlying data changes, or displaying data lineage for AI/ML models? "Are there any templates, guidance, or AIsupport to document ML/AI models and applications, such as automatic documentation suggestions when the platform recognizes a model? Leading-edge: "Does it allow the implementation of enterprise governance frameworks for end-to-end oversight, enabling continuous compliance monitoring and dynamic risk assessments linked to changing data inputs?" #data #datacatalog #ai #aiml