June 2023

These features and Azure Databricks platform improvements were released in June 2023.

Databricks Terraform provider updated to version 1.20.0

June 29, 2023

Databricks Terraform provider version 1.20.0 adds support for the Unity Catalog databricks_catalog_workspace_binding resource, improves the export of users and groups at the account level, and more. For details, see the changelog for version 1.20.0.

Databricks CLI updated to version 0.200.1 (Public Preview)

June 28, 2023

Databricks CLI version 0.200.1 updates the fs ls and fs cp commands, supports local files as the default for fs commands, adds provider detection to the repos create command, and more. For details, see the changelog for version 0.200.1.

Databricks SDK for Go updated to version 0.12.0 (Beta)

June 28, 2023

Databricks SDK for Go version 0.12.0 adds support for job-level parameters and parameter definitions, job task conditional execution, and several bug fixes for other services. For details, see the changelog for version 0.12.0.

Databricks SDK for Go updated to version 0.11.0 (Beta)

June 28, 2023

Databricks SDK for Go version 0.11.0 introduces several breaking changes to the Go APIs, including dozens of additions, renames, and consolidations. For details, see the changelog for version 0.11.0.

Databricks SDK for Python (Beta)

June 28, 2023

The Databricks SDK for Python is now available as a Beta. The Databricks SDK for Python enables you to automate Azure Databricks accounts, workspaces, and resources by running Python code.

See the Databricks SDK for Python.

Databricks SDK for Go (Beta)

June 28, 2023

The Databricks SDK for Go is now available as a Beta. The Databricks SDK for Go enables you to automate Azure Databricks accounts, workspaces, and resources by running Go code.

See the Databricks SDK for Go.

Access audit log, billable usage, and lineage system tables (Public Preview)

June 28, 2023

Account admins can now enable access to system tables in Azure Databricks accounts. System tables are an Azure Databricks-hosted analytical store of an account’s operational data. System tables provide you with easily-accessed account-wide observability data.

For more information, see Monitor usage with system tables.

Models in Unity Catalog (Public Preview)

June 28, 2023

You can now upgrade MLflow Model Registry workflows to govern models through Unity Catalog. Unity Catalog provides centralized access control, auditing, lineage, model sharing across workspaces, and better MLOps deployment workflows. Databricks recommends using Models in Unity Catalog instead of the existing workspace model registry, which will be deprecated in the future. See Manage model lifecycle in Unity Catalog for details.

Databricks Marketplace is now GA

June 27, 2023

You can now use Databricks Marketplace to discover, access, and deploy hundreds of data products and notebooks from top providers into your AI and analytics workflows. Highlights since the beginning of the public preview include new providers—including Dun and Bradstreet, LexisNexis, Refinitiv (London Stock Exchange Group), YipitData, and ZoomInfo—along with the addition of private exchanges and simplified consumer permission management using the USE_MARKETPLACE_ASSETS privilege.

See What is Databricks Marketplace?.

Databricks Runtime 13.2 (Beta)

June 26, 2023

Databricks Runtime 13.2 and Databricks Runtime 13.2 ML are now available as Beta releases.

See Databricks Runtime 13.2 (EoS) and Databricks Runtime 13.2 for Machine Learning (EoS).

Databricks extension for Visual Studio Code (General Availability)

June 26, 2023

The Databricks extension for Visual Studio Code is now generally available. Connect to the lakehouse, perform interactive debugging, and run Jupyter notebooks, all without leaving your IDE. Use your IDE’s code navigation, refactoring, completion, and error identification.

See the Databricks extension for Visual Studio Code.

Enable customer-managed keys using Azure CLI and Powershell without a template

June 23, 2023

You can now enable customer-managed keys for managed services and managed disks using either Azure CLI and Powershell without using a template. This was already available for customer-managed keys for DBFS. See Enable customer-managed keys for managed services, Configure customer-managed keys for Azure managed disks, and Customer-managed keys for DBFS root.

Serverless SQL warehouses work with back-end Azure Private Link enabled

June 22, 2023

Enabling back-end Azure Private Link no longer prevents serverless SQL warehouses from starting. Serverless compute resources don’t use the back-end Azure Private Link connection if it’s enabled. See Enable serverless SQL warehouses and Enable Azure Private Link back-end and front-end connections.

See a visual overview of completed job runs in the Azure Databricks Jobs UI

June 22, 2023

An enhanced Azure Databricks Jobs UI provides a visual overview of completed job runs, filterable by run status and time, and a list of the most common job errors. The default time filter covers the previous 48 hours. See View recent job runs.

Databricks CLI (Public Preview)

June 21, 2023

The Databricks command-line interface (Databricks CLI) has undergone a major overhaul. The new CLI covers all Databricks REST API operations and supports all Azure Databricks authentication types. macOS and Linux users can install the new CLI with Homebrew. Windows is also supported.

Improved pagination of results from List all jobs and List job runs API requests

June 14, 2023

You can now use the page_token parameter to page the results returned from the GET /api/2.1/jobs/list and GET /api/2.1/jobs/runs/list Jobs API requests. This token-based pagination replaces the deprecated offset and limit parameters and improves the service scalability for these requests.

Full-page workspace browser includes Repos

June 14, 2023

The full-page workspace browser experience now extends to Repos. You can browse content in Databricks Repos alongside your workspace content by clicking Workspace in the sidebar. You can also browse content from within a notebook using a contextual browser. For more information about the full-page workspace browser, see Navigate the workspace and Workspace browser.

Databricks Terraform provider updated to version 1.19.0

June 14, 2023

Databricks Terraform provider 1.19.0 adds run_as and run_as_user_name to databricks_job, adds more attributes to databricks_cluster_policy, adds force_destroy to databricks_external_location, and more. For more details, see the changelog for version 1.19.0.

Run jobs as a service principal (Public Preview)

June 13, 2023

You can now grant Azure Databricks users, service principals, and groups permissions to use a service principal. This allows users to run jobs as the service principal, instead of running as the job owner’s identity, to prevent jobs from failing if a user leaves your organization or a group is modified.

See Roles for managing service principals and Configure identity for job runs.

New service principal UI provides better management experience

June 13, 2023

You can now use the new service principal UI to manage service principals in your workspace. The service principal UI lets you add service principals to your workspace, configure entitlements, and manage service principal permissions. See Manage service principals.

Home folders restored when users are re-added to workspaces

June 9, 2023

When an admin removes a user from a workspace and then re-adds them, the user’s home folder is now restored. Previously, upon being re-added to the workspace, the user received a new empty home folder. Their old home folder was retained as a backup. For more information about workspace folders, see Organize workspace objects into folders.

Databricks Marketplace: Private exchanges are now available

June 7, 2023

Private exchanges enable data providers to share data products with a select group of invited consumers. Private listings appear on the invited consumers’ Marketplace home page, but do not appear in the public marketplace. See Create and manage private exchanges in Databricks Marketplace.

Databricks Marketplace: Consumers can uninstall data products using the UI

June 5, 2023

Data consumers can now uninstall any data product using the Marketplace My requests interface. See Delete installed data products.

Databricks Marketplace: providers can create their own profiles

June 5, 2023

Data providers can now create and update their own profiles, once their provider application is approved. See Create your Marketplace provider profile.

Databricks Connect V2 is GA for Python

June 2, 2023

Beginning with Databricks Runtime 13.1, Databricks Connect is now generally available for Python. This release adds support for Structured Streaming, all Delta features, and Pandas on Spark. It also includes support for OAuth (Experimental), by relying on the included Databricks SDK for Python.

Databricks Terraform provider updated to version 1.18.0

June 2, 2023

Databricks Terraform provider 1.18.0 adds a databricks_volume resource, databricks_model_serving support to databricks_permissions, and more. For more details, see the changelog for version 1.18.0.

New Databricks Marketplace providers

June 1, 2023

Eight new providers have published listings in Databricks Marketplace in the past month. These include:

• IQVIA: Healthcare and life sciences data.
• Morningstar: Investment offering data and research insights such as ESG risk rating data.
• Vaisala: Weather and environment data.
• Sucali: Real estate performance metrics for hedge funds and investors.
• Veritas Data Research: Comprehensive index of US mortality data from 2010 to present.
• Collectors Data Store: Sports and trading card data for advertisers and consumers.
• Tego Cyber: Cybersecurity and threat intelligence data
• Ribbon Health: Healthcare provider data.

If you have questions about these providers or Marketplace, reach out to dataconsumers@databricks.com. See also What is Databricks Marketplace?.

Databricks Runtime 13.1 is GA

June 1, 2023

Databricks Runtime 13.1 and Databricks Runtime 13.1 ML are now generally available.

See Databricks Runtime 13.1 (EoS) and Databricks Runtime 13.1 for Machine Learning (EoS).

View data from notebooks, SQL editor, and Catalog Explorer with the unified schema browser (Public Preview)

June 1, 2023

The new unified schema browser lets you view all of the data in the Unity Catalog metastore without leaving a notebook or the SQL editor. You can select For you to limit the view to only active data you’re using and your favorites.

For details about Catalog Explorer, see Catalog Explorer.

For information about accessing Catalog Explorer from a notebook, see Browse data.