Knowledge Catalogs Vs. Metadata Catalogs: What’s the Distinction?

Knowledge Catalogs Vs. Metadata Catalogs: What’s the Distinction?
Knowledge Catalogs Vs. Metadata Catalogs: What’s the Distinction?


Knowledge catalogs and metadata catalogs share some similarities, notably of their practically similar names. And whereas they’ve some frequent capabilities, there are additionally necessary variations between the 2 entities that massive knowledge practitioners ought to find out about.

Metadata catalogs, that are typically referred to as metastores or technical knowledge catalogs, have been within the information currently. In case you’re an everyday Datanami reader (and we definitely hope you might be!), you’d have learn loads metadata catalogs on the Snowflake and Databricks conferences final month, when the 2 opponents dedicated to open sourcing their respective metadata catalogs, Polaris and Unity Catalog.

So what’s a metadata catalog, and why do they matter? (We’re glad you requested!) Learn on to study extra.

Metadata Catalogs

A metadata catalog is outlined because the place the place one shops the technical metadata describing the info you have got saved as a tabular construction in an information lake or a lakehouse.

Essentially the most generally used metadata catalog is the Hive Metastore, which was the central repository for metadata describing the contents of Apache Hive tables. Hive, after all, was the relational framework that allowed Hadoop customers to question HDFS-based knowledge utilizing good previous SQL, versus MapReduce.

Hive and the Hive Metastore are nonetheless round, however they’re within the means of being changed by a more recent technology of know-how. Desk codecs, similar to Apache Iceberg, Apache Hudi, and Databricks Delta Desk, carry many benefits over Hive tables, together with assist for transactions, which boosts the accuracy of information.

These desk codecs additionally require a technical layer–the metadata catalog–to assist customers know what knowledge exists within the tables and to grant or deny entry to that knowledge. Databricks helps this perform in its Unity Catalog. For Iceberg, merchandise similar to Project Nessie, which was developed by engineers at Dremio, sought to be the “transactional catalog” brokering knowledge entry to numerous open and business knowledge engines, together with Hive, Dremio, Spark, and AWS Athena (primarily based on Presto), amongst others.

Snowflake developed and launched (or pledged to launch, anyway) Polaris to be the usual metadata catalog for the Apache Iceberg ecosystem. Like Nessie, Polaris makes use of Iceberg’s open REST-based API to get entry to the descriptive metadata of the Parquet knowledge that Iceberg shops. This REST API then serves because the interface between the info saved in Iceberg tables and knowledge processing engines, similar to Snowflake’s native SQL engine in addition to a wide range of open-source engines.

Knowledge Catalogs

Knowledge catalogs are sometimes third-party instruments that corporations use to prepare the entire knowledge they’ve saved throughout their organizations. They sometimes embody some facility that enables customers to seek for knowledge their group could personal, which implies knowledge catalogs typically have some knowledge discovery element.

Many knowledge catalogs, similar to Alation’s catalog, have additionally advanced to incorporate entry management performance, in addition to knowledge lineage monitoring and governance capabilities. In some circumstances, knowledge administration software distributors that started off offering knowledge governance and entry management, similar to Collibra, have advanced the opposite manner, to additionally embody knowledge catalogs and knowledge discovery capabilities.

And like metadata catalogs, common knowledge catalogs–or what some within the business time period “enterprise” knowledge catalogs–are additionally absolutely concerned in gobbling up metadata to assist them monitor varied knowledge property. One enterprise knowledge catalog vendor, Atlan, focuses its efforts on unifying the metadata generated by completely different datasets and synchronizing them by a metadata “management airplane,” thereby making certain that the enterprise metrics don’t get too out of whack.

By now, you’re most likely questioning “So what the heck is the distinction?! They each monitor metadata, they usually each have “knowledge catalog” of their identify. So what’s the distinction between a metadata catalog and an information catalog.

So What’s The Distinction?!

To assist us decode the variations between these two catalog varieties, Datanami lately talked to Felix Van de Maele, the CEO and co-founder of Collibra, one of many main knowledge catalog distributors within the massive knowledge house.

“They’re very various things,” Van de Maele mentioned. “If you concentrate on Polaris catalog and Unity Catalog from Databricks–and AWS and Google and Microsoft all have their catalogs–it’s actually this concept that you simply’re in a position to retailer your knowledge anyplace, on any clouds…And I can use any form of knowledge engine like a Databricks, like a Snowflake, like a Google, AWS, and so forth, to eat that knowledge.”

However what Collibra and different enterprise knowledge catalogs do is sort of completely different, Van de Maele mentioned.

Felix Van de Maele is the CEO and co-founder of Collibra

“What we do is we offer far more of the enterprise context,” he mentioned. “We offer what we name that information graph, that enterprise context the place you’re truly defining and managing your insurance policies. Insurance policies similar to what’s the standard of my knowledge? What enterprise guidelines does my knowledge must comply to? What privateness insurance policies does my knowledge must comply to? Who must approve it? How will we seize attestations? How will we do certification? How do I construct a enterprise glossary with enterprise phrases and clear definitions?

“That’s very completely different than a Polaris catalog on high of Iceberg that’s the bodily metadata. And that’s an actual differentiation,” he mentioned.

Van de Maele helps the open knowledge lakehouse structure that has emerged, which provides prospects the liberty to retailer their knowledge in open desk codecs, similar to Iceberg, Delta, and Hudi, and question it with any engine. His prospects, a lot of that are Fortune 500 enterprises, retailer knowledge throughout many knowledge platforms and use the Collibra Knowledge Intelligence platform to assist management and govern entry to that knowledge.

Completely different Roles

Prospects ought to perceive that, whereas the names are comparable, metadata catalogs and knowledge catalogs play very completely different roles.

“The way in which I differentiate between the 2 is we do coverage definition and administration, they do coverage enforcement,” Van de Maele mentioned. “And truly I feel that’s the fitting structure.”

(Den Rise/Shutterstock)

The metadata catalogs sometimes would not have performance to permit customers to arrange enterprise insurance policies round knowledge entry. As an example, they received’t allow you to arrange entry controls to allow a advertising and marketing workforce to entry all buyer knowledge aside from something that’s been marked “labeled,” during which case it should be masked, Van de Mael mentioned.

“We will have advertising and marketing knowledge in Databricks, we’ve got advertising and marketing knowledge in Salesforce, we’ve got advertising and marketing knowledge in Google, and anyplace persons are utilizing advertising and marketing knowledge, I must ensure that the fitting knowledge is classed and masked,” he mentioned. “So we push that down in Databricks, in Snowflake, in Google, in Amazon and in Microsoft.”

Prospects might outline their very own knowledge entry insurance policies with no software like Collibra’s, Van de Mael mentioned. In any case, it’s simply SQL on the finish of the day. However then they would want another methodology to maintain monitor of the tens of millions of columns unfold throughout varied knowledge platforms. Offering perception into what knowledge exists and the place, after which making certain prospects are accessing it in keeping with the corporate’s governance guidelines, is the position Collibra serves.

On the identical time, Collibra relies upon metadata catalogs for the enforcement mechanisms. Different enforcement mechanisms have been tried, similar to proxies and drivers, Van de Maele mentioned, however none of it really works.

“We expect the metadata catalog strategy with open desk format is definitely the fitting strategy,” he mentioned. “We wish to have these knowledge platforms be capable to do this natively, in any other case scalability and efficiency at all times turn into an issue.”

Databricks Unity Catalog seems to be the exception right here. Unity Catalog, which Databricks just open sourced last month, supplies the low-level management over technical metadata in addition to higher-level capabilities, similar to knowledge governance, entry management, auditing, and lineage. In that respect, Unity Catalog seems to compete with the enterprise knowledge catalog distributors.

Associated Gadgets:

What the Big Fuss Over Table Formats and Metadata Catalogs Is All About

Databricks to Open Source Unity Catalog

What to Look for in a Data Catalog

Leave a Reply

Your email address will not be published. Required fields are marked *