What is Databricks AI/BI?

  • Article

Databricks AI/BI is a new type of business intelligence product designed to provide a deep understanding of your data’s semantics, enabling self-service data analysis for everyone in your organization. AI/BI is built on a compound AI system that draws insights from the full lifecycle of your data across the Azure Databricks platform, including ETL pipelines, lineage, and other queries.

How does AI/BI work?

Compound AI systems combine multiple AI technologies or models to solve complex problems. Instead of relying on a single AI model or algorithm, a compound AI system integrates multiple interacting components to enhance performance and accuracy.

For example, in a system analyzing customer feedback, natural language processing can be used to understand text sentiment, while machine learning models can predict customer behavior based on this sentiment. By leveraging these complementary AI technologies, compound AI systems can deliver more accurate and insightful results than the individual AI models alone.

Traditional BI tools have relied on reports and dashboards, often requiring extensive involvement from data professionals to create new visualizations. While AI assistants have been integrated into BI tools to address this issue, they frequently struggle with real-world data complexities, providing impressive demos but failing in practice.

The semantic model for an organization often relies on the knowledge held by those who work with the data. Databricks AI/BI captures this understanding from interactions across Databricks. It augments the existing context in the Data Intelligence Platform, leveraging this knowledge to provide practical, real-world answers. It powers two complementary product experiences:

  • AI/BI Dashboards: an AI-powered, low-code dashboarding solution that includes all the conventional BI capabilities for answering a fixed set of business questions.
  • Genie: a conversational interface that can learn the underlying data and semantics continuously based on human feedback. It can answer a much broader set of business questions based on its reasoning capabilities, while still providing trusted assets for query patterns specified by the data teams. Genie is in Public Preview.

AI/BI dashboards

Dashboards remain an effective method for operationalizing predefined analytics for regular use. AI/BI Dashboards streamline this process with an AI-powered, low-code authoring experience that simplifies data and visualization configuration. These dashboards include standard BI capabilities such as visualizations, cross-filtering, and periodic PDF snapshots via email. They do not require semantic models, data extracts, or additional management services. See Dashboards.

AI/BI Genie

To address a wide range of questions not covered by dashboards, AI/BI offers a conversational interface called Genie. Unlike traditional fixed-chart tools, Genie can learn from the underlying data and respond to user queries with adaptable visualizations and suggestions. It seeks clarification and provides alternative solutions as needed. Genie is designed to improve over time through human feedback, offering tools for analysts to verify and refine its outputs. Features like instructions, trusted assets, confidence voting, and quality monitoring enable data teams to fine-tune Genie’s performance and ensure reliable responses.

Genie uses a system of tools, such as trusted assets, to enhance trustworthiness. Analysts can integrate trusted logic from sources like Unity Catalog into Genie, allowing it to answer questions with validated logic and transparently share the trusted status of the provided answers.

See Work with an AI/BI Genie space to learn how to enable and use Genie in your workspace.

Platform integration

Databricks AI/BI is built on top and tightly integrated with the Databricks Data Intelligence Platform. Key features of AI/BI include:

  • Unified governance and lineage: AI/BI is seamlessly integrated into Unity Catalog, aligning with its governance framework and adhering to any global policies set by administrators. Through Unity Catalog’s lineage visualization, data producers or administrators can track the usage of their data assets within AI/BI. This traceability back to the dataset’s ingestion instills confidence in the analysis results.
  • Seamless sharing, without new user licenses: AI/BI is built into the Databricks IAM platform, which integrates directly with many popular IDPs so that you can share your analysis with anyone in your organization. Databricks AI/BI does not have seat-based restrictions, so you can add anyone from your organization without the expense of adding new licenses.
  • Industry-leading price/performance: AI/BI is tightly integrated with SQL warehouses and the Photon engine, which contain unique optimizations to deliver high-performance interactions. You can expect optimal performance across a wide range of data volumes, from megabytes to petabytes.
  • No data extraction required: AI/BI integrates with your existing data ecosystem, eliminating the need to extract datasets to a separate BI engine. This integration improves data freshness and simplifies data governance, leading to a more streamlined data analysis process.

Next steps

Choose a tutorial to build a dashboard. See Dashboard tutorials.