June 2021

These features and Azure Databricks platform improvements were released in June 2021.

Correction: Repos for Git is enabled by default in new and existing workspaces in some regions

June 30, 2021

In a release note for June 2, 2021, Databricks announced that Repos for Git is enabled by default in new workspaces in some regions, but omitted the following important detail: Repos for Git is also enabled in existing workspaces if Notebook Git Versioning is already enabled. The original release note has been updated.

Change to Feature Store permissions

June 28 - July 6, 2021: Version 3.49

Users with CAN_VIEW_METADATA permission on a feature table can now publish it to online stores.

Improved access to results in the MLflow runs table

June 28 - July 6, 2021: Version 3.49

Changes to the MLflow runs table provide quicker access to results.

• The Metrics columns now appear before the Parameters columns, making it easier to identify the best run.
• The Start Time column now shows the age of the run, not the start time. This reduces the width of the column and allows more information to be visible without horizontal scrolling. To display the start time, hover over the value in the column.

Better cost visibility for Delta Live Tables

June 28 - July 6, 2021: Version 3.49

When you create a new pipeline or view the Pipeline Details UI, you will now see the DBU per hour for the Delta Live Tables default cluster.

Enhanced data quality constraints for Delta Live Tables

June 28 - July 6, 2021: Version 3.49

You can now use the expect_all_or_fail, expect_all_or_drop, and expect_all Python APIs to define multiple data quality constraints on Delta Live Tables datasets. These APIs are useful when applying a library of common expectations to one or more datasets in a pipeline. For more information, see Manage data quality with Delta Live Tables.

API changes for updating and replacing IP address lists

June 28 - July 6, 2021: Version 3.49

A new PATCH operation for /ip-access-lists/<list-id> in the Databricks REST API 2.0 allows you to programmatically update the individual values in an existing IP address list. For more information, see Update an IP access list.

The existing PUT operation for /ip-access-lists/<list-id> in the Databricks REST API 2.0 allows you to programmatically replace the contents of an existing IP address list while maintaining the list’s original internal ID. For more information, see Replace an IP access list.

Databricks ODBC driver 2.6.17

June 22, 2021

We have released version 2.6.17 of the Databricks ODBC driver (download). This release adds support for Cloud Fetch, which is a mechanism for fetching data in parallel through cloud storage to bring the data faster to BI tools. This release also reduces latency by removing unnecessary roundtrips between the client and the Databricks Runtime. See How We Achieved High-bandwidth Connectivity With BI Tools.

Databricks Runtime 7.5 series support ends

June 16, 2021

Support for Databricks Runtime 7.5, Databricks Runtime 7.5 for Machine Learning, and Databricks Runtime 7.5 for Genomics ended on June 16. See Databricks support lifecycles.

Optimize performance and control costs by using different pools for the driver node and worker nodes

June 14-21, 2021: Version 3.48

You can now optimize your cluster by attaching it to a different pool for the driver node and worker nodes. For example, you can use a compute-optimized pool for the driver node and a storage-optimized pool for nodes. This also helps you control costs by assigning different tags to each instance pool. If you choose, you can use the same pool for drivers and worker nodes.

For more information, see Pool configuration reference.

Registry-wide permissions for Model Registry

June 14-21, 2021: Version 3.48

You can now set default model permissions for all models in a workspace’s Model Registry. Permissions set at this level apply to all new and existing models registered in Model Registry in that workspace. For more information, see MLflow model ACLs.

A user’s home directory is no longer protected when you delete a user using the SCIM API

June 14-21, 2021: Version 3.48

When you delete a user using the SCIM API, the user’s home directory is no longer protected, and a workspace administrator can delete it. This behavior matches what happens when you delete a user using the admin console.

Databricks Runtime 8.3 and 8.3 ML are GA; 8.3 Photon is Public Preview

June 8, 2021

Databricks Runtime 8.3 and 8.3 ML are now generally available. 8.3 Photon is in Public Preview.

For information, see the full release notes at Databricks Runtime 8.3 (EoS) and Databricks Runtime 8.3 for ML (EoS).

Python and SQL table access control (GA)

June 8, 2021

Table access control lets you programmatically grant and revoke access to your data using the Azure Databricks view-based access control model. Table access control is available in two versions: SQL-only and Python and SQL. Previously only SQL-only table access control was generally available. Now Python and SQL table access control is generally available. See Enable Hive metastore table access control on a cluster (legacy).

Jobs UI and API now show the owner of a job run

June 2-7, 2021: Version 3.47

The jobs UI and API now include the user that a job will run as, making it easier to understand, debug, and audit job permissions. This is a read-only value set to the job creator if job access control is disabled or the job’s is_owner permission if job access control is enabled. For more information about managing job permissions, see Control access to a job.

Protect sensitive Spark configuration properties and environment variables using secrets (Public Preview)

June 7, 2021

You can protect sensitive Spark configuration properties and environment variables using secrets. Retrieved secrets are redacted from notebook output and Spark driver and executor logs. See Secrets.

If you currently use plaintext passwords in a Spark configuration, Databricks recommends that you rotate your passwords and use secret paths in the Spark configuration.

Repos for Git is enabled by default in new and existing workspaces in some regions

June 2, 2021

This release note was updated on June 30, 2021.

For new workspaces and for existing workspaces with Notebook Git Versioning enabled, Repos for Git is now enabled by default in some regions. Default Repos enablement will be rolled out to more regions in the coming months. To learn how to enable or disable Repos yourself, and to verify the settings for your workspace, see Git integration for Databricks Git folders.

Redesigned Workspace Settings UI

June 2-7, 2021: Version 3.47

The new Workspace Settings tab in the Admin Console replaces the Workspace Storage, Access Control Lists, and Advanced tabs for streamlined workspace configuration.

Updates to ListTokens and ListAllTokens database queries expired tokens

June 2-7, 2021: Version 3.47

Database queries ListTokens or ListAllTokens no longer return expired tokens. Indefinite tokens (those with expiryTime set to -1) are returned.

Confirmation now required when granting or revoking Admin permissions

June 2-7, 2021: Version 3.47

A confirmation dialog now appears when an Admin user grants or revokes Admin privileges in the web UI.

Changes to keyboard shortcuts in the web UI

June 2-7, 2021: Version 3.47

In a notebook, the way to open the list of keyboard shortcuts has changed. To open the list, click the icon.