In an exciting development at the recent Data + AI Summit in San Francisco, Databricks announced that it is open-sourcing Unity Catalog, its powerful metadata catalogue tool. This move marks a significant shift in the data management landscape, promising to enhance interoperability and innovation within the industry.
Unity Catalog: The “USB” for Data Access
Ali Ghodsi, CEO of Databricks, described Unity Catalog as the “USB” for data access during his keynote address. He explained, “All the silos that you had before, they can just access one copy of the data that’s in a standardised USB format under your ownership. It goes through one governance layer that’s just standardised–that’s Unity Catalog–for all of your data.”
Unity Catalog’s open sourcing is set to standardise data access and security, supporting multiple data formats like Delta, Apache Iceberg, and Apache Hudi via Databricks’ Delta Lake UniForm format. By supporting Iceberg’s REST-based API, Databricks aims to create a unified, open data environment where all systems can work together seamlessly.
A New Era of Open Lakehouses
This move by Databricks reflects a strategic response to the rising popularity of Apache Iceberg and the demand for open lakehouse platforms. By opening up Unity Catalog, Databricks ensures that data management is more flexible and accessible, paving the way for greater innovation across the industry.
Suranga Fernando, Co-Founder of Arreoblue, shared his insights on this pivotal development, commenting: “Databricks’ decision to open-source Unity Catalog is a landmark moment for the data community. We have seen first-hand how tough it is for large organisations to avoid the problems that come with siloed data, and how Unity can help. Increased use of open standards for data access, security and governance will help to eliminate these silos and empower companies to innovate more freely. This update has sent a ripple of excitement through the team as it aligns perfectly with Arreoblue’s mission to harness the full potential of data through open and interoperable systems.”
Industry Support and Future Prospects
The announcement has also been met with enthusiasm from major industry players like AT&T and Nasdaq. Matt Dugan, AT&T’s vice president for data platforms, stated, “With the announcement of Unity Catalog’s open sourcing, we are encouraged by Databricks’ step to make lakehouse governance and metadata management possible through open standards.” Lenny Rosenfeld from Nasdaq echoed these sentiments, highlighting the potential for improved governance and data application modernisation.
Databricks plans to release Unity Catalog on GitHub soon, reinforcing their commitment to open-source development. While the specific open-source foundation for Unity Catalog is yet to be decided, previous projects by Databricks have been managed by The Linux Foundation, suggesting they may forge a similar path.
At Arreoblue, we see Databricks’ open-sourcing of Unity Catalog as a transformative move that will drive the future of data management. By creating a universal standard for data access and security, Databricks is setting the stage for unprecedented innovation and collaboration in the data community. Looking ahead, we are excited to see how those within the industry leverage these advancements to deliver greater value to the data management space.
Stay tuned to Arreoblue for more insights and updates on the latest in data technology.
About Arreoblue
Arreoblue is a forward-thinking solutions provider specialising in tailor-made strategies to optimise business processes and foster growth. With our Assess, Accelerate and Amplify methodology, our experts will utilise their decades of experience to empower your people with a platform of success and solution that works FOR you.
To find out more, get in touch with one of our dedicated team today at info@arreoblue.com