ALI GHODSI, CO-FOUNDER AND CEO DATABRICKS, SUMMARIZING ANNOUNCEMENTS AT DATA AND AI SUMMIT, 2025

Originally Published June 2025, on ARCweb.com by Colin Masson

It’s been a busy few months on my voyage of discovery into the evolving world of Data Fabrics. As I’ve worked to define what constitutes a truly “Industrial-grade Data Fabric” for this new era of AI, one name has appeared on the horizon with such remarkable frequency: that I posed the question “Is Databricks the North Star for Industrial AI?”, in an earlier blog post! From partnerships with industrial software leaders to alliances with enterprise application titans, their presence has been a constant tailwind. It seems fitting, then, that the final leg of this journey before I retreat to finalize my ARC Advisory Group reports was a trip to San Francisco for the Databricks Data & AI Summit.

The summit was Databricks’ most audacious move yet to not just participate in the data and AI market, but to define its very architecture. The message was clear: the company is evolving beyond a unified analytics platform and is positioning its Data Intelligence Platform as a vertically integrated operating system for enterprise AI. It’s an ambitious play, and for those of us focused on the industrial sector, it raises a critical question: Is this the “North Star” that can guide industrial organizations through the complexities of digital transformation?

A Legacy of Innovation: Setting the Stage

To understand the magnitude of the 2025 announcements, it’s important to remember the foundational innovations that fueled Databricks’ ascent. Their strategy has always been to tackle the biggest challenges in data and AI, resulting in a series of paradigm-shifting contributions:

• Pioneering Apache Spark: At its core, Databricks leveraged the open-source power of Apache Spark, a world-class distributed processing engine, to handle massive datasets at speed.
• Inventing the Lakehouse Architecture: Databricks challenged the traditional separation of data lakes and data warehouses, creating the “Lakehouse”. This architecture, built on open formats like Delta Lake, allows a single system to handle everything from data warehousing and SQL analytics to real-time AI workloads, a key market differentiator.
• Unifying Governance with Unity Catalog: Recognizing that a powerful platform is useless without control, Databricks introduced Unity Catalog. It has become the governance “center of gravity” for the entire platform, providing a unified catalog for all data and AI assets, including robust data lineage and discovery features.
• Championing a Unified Platform: From the beginning, the core narrative articulated by CEO Ali Ghodsi has been to solve the fragmentation of the data and AI estate. The vision has always been a single, unified platform where the entire lifecycle—from data ingestion to model deployment—can occur within one architecture.

This history of tackling fundamental architectural problems is the bedrock upon which the latest wave of innovation was built.

A Barrage of Innovation: Unpacking the Summit’s Big Bang

The central theme was that the fragmentation of data and AI tools remains the primary obstacle to enterprise success. The flurry of announcements was positioned as the definitive solution. For those of us tracking the space, the announcements felt less like individual product launches and more like the interconnected components of a new enterprise operating system.

Here’s a more detailed breakdown of the significant launches:

• Lakebase: This marks Databricks’ strategic entry into the operational database market. It’s a new, fully-managed PostgreSQL database built on a lakehouse architecture with separated storage and compute. Designed for low-latency transactional (OLTP) workloads, its purpose is to unify OLTP and OLAP systems to power real-time, agent-driven AI applications. Features like the ability to programmatically “branch” a database for experimentation are tailored for autonomous agents, not traditional DBAs.
• Agent Bricks: Moving beyond the hype, this is a UI-driven framework designed to productize the creation of trustworthy, domain-specific AI agents. It abstracts away difficult tasks by including automated testing, cost optimization, and built-in “judge” models to evaluate agent performance. The system even incorporates “LLM judges” and automated evaluators, creating a feedback loop where agents can be tested and optimized by other agents. This is central to moving AI agents from proof-of-concept to governed, production-ready systems.
• Lakeflow: Aiming to consolidate the fragmented ETL/ELT tool market, Lakeflow offers a unified solution for all data engineering needs. It combines no-code connectors, declarative transformations for building pipelines, and AI-assisted authoring for both batch and real-time data streams. A key component is Lakeflow Designer, a visual, no-code interface that allows business analysts and other less technical users to build ETL pipelines using drag-and-drop and natural language, democratizing data engineering.
• Unity Catalog Enhancements: The platform’s governance core received significant updates to reinforce the “open but better on Databricks” strategy. Key among these is full read/write support for the open Apache Iceberg format. Other major additions include Unity Catalog Metrics and a curated internal data marketplace to help business users discover certified data assets, along with new governance features like Attribute-Based Access Control (ABAC) and automated data classification.
• AI/BI Genie & Databricks One: This is a direct push to democratize data access for non-technical users. It features a redesigned user experience called Databricks One, which includes the now generally available AI/BI Genie, a natural language query tool for generating instant insights.
• MLflow 3.0: A major update to the popular open-source MLOps framework, MLflow 3.0 has been redesigned specifically for the challenges of Generative AI. It introduces critical features for agent observability, prompt versioning, and cross-platform monitoring to manage complex, multi-step AI systems.
• Lakebridge: This is a direct and practical appeal to IT leaders burdened by legacy data warehouses. Lakebridge is a free, AI-powered tool designed to automate and accelerate migrations from competitors like Teradata. By handling tasks like SQL conversion and validation, it aims to de-risk modernization initiatives.

The Industrial Playbook: Power Through Partnership

While the technology is impressive, what truly defines Databricks’ industrial strategy is its deep understanding that it cannot go it alone. Instead of trying to build a monolithic “industrial cloud,” Databricks is positioning itself as the indispensable analytics engine at the center of a vibrant ecosystem. The success of its vision, especially in the complex industrial world, hinges on strategic partnerships.

This ecosystem-driven approach is a core pillar of their industrial offensive:

• Microsoft: The foundational partnership for scale. Azure Databricks is a first-party Microsoft service, ensuring tight integration and a streamlined experience for customers. For industrial IoT, this is critical, providing a secure and scalable path from edge sensors—via services like Azure IoT Hub—to advanced AI in the cloud.
• AVEVA: Bridging the IT/OT divide. Named the 2025 Databricks Manufacturing ISV Partner of the Year, the AVEVA partnership is a textbook ecosystem strategy. AVEVA’s CONNECT platform handles the “first-mile” of collecting and contextualizing complex data from industrial control systems, which is then seamlessly shared with Databricks for large-scale analytics and AI.
• SAP: Unlocking the enterprise core. This landmark partnership is potentially transformative. By integrating Databricks natively into SAP’s Business Data Cloud, the collaboration solves a decades-old challenge: freeing mission-critical data from SAP systems for use in modern AI applications. For manufacturers, this allows them to combine ERP data with real-time OT data from the factory floor in a single, governed environment.
• Kinaxis: Powering intelligent supply chains. The integration with Kinaxis, a leader in supply chain orchestration, extends Databricks’ reach into another mission-critical domain. By leveraging Databricks’ data infrastructure, Kinaxis can help customers build more resilient and agile supply chains that can react faster to disruption.

Driving Industrial Value: From Outcome Maps to Business Outcomes

Databricks has mounted a significant offensive to capture the industrial AI market. The strategy is to become the central data and AI platform upon which a broader industrial ecosystem operates, solving key challenges by providing a powerful analytics engine. This was on full display at the summit, with a clear focus on enabling tangible business outcomes.

To accelerate adoption, Databricks is moving beyond just providing tools to offering blueprints. The company is now rolling out industry-specific “Data Intelligence Outcome Maps”. These are frameworks designed to guide customers through the most valuable use cases in their sector, such as:

• Predictive Maintenance
• Supply Chain Optimization
• Quality Control
• Energy Efficiency

These Outcome Maps are backed by tangible “Solution Accelerators” available on GitHub, which are pre-built, functional notebooks for use cases like automotive geospatial analysis or ESG scoring, significantly lowering the barrier to implementation.

Marquee customer presentations underscored this industrial push. A compelling session from Rivian detailed how the electric vehicle manufacturer is building a “battery data fabric” on the Databricks platform. They are unifying data from the entire battery lifecycle—from cell production and pack assembly to in-field vehicle performance—to improve analytics, enhance performance, and predict battery health. This is a prime example of using a unified data and AI platform to tackle one of the most complex and data-intensive challenges in modern manufacturing.

Resonating Across the Enterprise: A Platform for Everyone?

Databricks has skillfully designed its platform to appeal to the different needs of key enterprise personas, from the server room to the plant floor and the front office. It is not a one-size-fits-all platform; it is a portfolio of tailored experiences that all connect to the same governed data backend.

• For IT Leaders, the message is about simplification and governance. Unifying analytical and transactional systems promises to reduce architectural complexity and lower costs. Unity Catalog provides the robust, enterprise-grade governance needed for compliance, while the commitment to open standards like Apache Iceberg offers an escape hatch from vendor lock-in.
• For Data Scientists, the platform is an accelerator. MLflow 3.0 has been redesigned for the generative AI era, offering essential tools for managing complex agentic systems. Agent Bricks promises to speed up the development of trustworthy AI agents, moving them from prototype to production with greater confidence.
• For OT Professionals, Databricks’ strategy is one of enablement through partnership. It doesn’t pretend to be an OT-native platform that connects directly to factory equipment. Instead, it relies on partners like AVEVA and Litmus for that “first mile” of data collection, positioning itself as the powerful analytics core where converged IT and OT data can be analyzed.
• For Mainstream Business Users, the goal is radical simplification and democratization. The new Databricks One interface and AI/BI Genie are specifically designed to democratize data access for business users. These tools allow non-technical users to query data using natural language, get instant insights, and even prototype changes in real-time, significantly boosting an organization’s AI fluency. The ultimate goal is a user experience that every employee can use to make better decisions with data.

The ARC Advisory Group Take: A Beacon for the Industrial AI Journey

After this deep dive at the Data & AI Summit, it’s clear why Databricks has become such a powerful beacon for data scientists, IT, OT, and business leaders alike. The company has laid out a clear-eyed and ambitious vision to become the de facto operating system for enterprise AI.

By benchmarking the platform against ARC’s framework for an Industrial-Grade Data Fabric, a deliberate and intelligent strategy comes into focus. Databricks is focused on delivering world-class native capabilities for the horizontal, domain-agnostic layers: data storage, metadata management, AI/ML development, and data processing. Simultaneously, it strategically cedes the vertical, industry-specific functions—like deep OT protocol connectivity and pre-built industrial applications—to its ecosystem of expert partners.

This isn’t a weakness; it is the very core of their strategy. It allows Databricks to do what it does best—provide scalable compute and state-of-the-art AI tooling—while leveraging the deep domain expertise of its partners. The result is a platform that is not just another vendor in the Industrial Data Fabric landscape, but rather aims to be the essential, horizontal foundation upon which all vertical Industrial Data Fabrics will be built.

Of course, the voyage ahead is not without peril. Competition is intense. Databricks faces a formidable challenge from Snowflake, which is pursuing a similar strategy, and from the major cloud hyperscalers. AWS, in particular, is a strong competitor with its own suite of industrial data services and deep relationships in the sector. Furthermore, as Databricks adds more products to serve everyone from engineers to business analysts, it faces the risk of its platform becoming overly complex. The powerful message of “democratization” must be consistently supported by a user experience that is genuinely simple and intuitive for these new personas.

Even so, Databricks continues to set the pace. By grounding the hype of agentic AI in the enterprise realities of governance and trust, and by building a powerful, partner-first coalition, they have plotted a compelling course. For industrial organizations navigating the often-turbulent waters of AI-driven transformation, Databricks has firmly established itself as a North Star worth watching.

Engage with ARC Advisory Group

For ARC Advisory Group recommendations for navigating the AI Wars, closing the digital divide by embracing Industrial AI, assembling your Industrial-grade Data Fabric, and modern Industrial AI technology stack – and governing and guiding major decisions about enterprise, cloud, industrial edge, and AI software, please contact Colin Masson at cmasson@arcweb.com or set up a meeting with me, or my fellow Analysts at ARC Advisory Group.

About the Author

Colin Masson

Reference:

Masson, C (2025). From Lakehouse to Powerhouse: How Databricks is Setting the Pace for Industrial AI. Available at: From Lakehouse to Powerhouse: How Databricks is Setting the Pace for Industrial AI | LinkedIn [Accessed: 14th July 2025].