April 2021

These features and Azure Databricks platform improvements were released in April 2021.

Databricks Runtime 8.2 (GA)

April 22, 2021

Databricks Runtime 8.2 and Databricks Runtime 8.2 ML are now generally available.

For information, see the full release notes at Databricks Runtime 8.2 (EoS) and Databricks Runtime 8.2 for ML (EoS).

Easier job management with the enhanced jobs user interface

April 20-26, 2021: Version 3.44

Databricks has redesigned the Schedule and orchestrate workflows user interface to make it easier to manage jobs. You can use the new job details page to perform all job related actions, including running, cloning, and deleting jobs.

Cluster policy changes are applied automatically to existing clusters at restart and edit

April 20-26, 2021: Version 3.44

It is now much easier to apply cluster policy changes to existing clusters. Any policy definition changes are applied to associated clusters at cluster restart and edit. Nonconforming clusters won’t be able to restart.

Track retries in your job tasks when task attempts fail

April 20-26, 2021: Version 3.44

You can now pass the task_retry_count parameter variable to a job task. The value of this variable is a count of the attempts to retry the task when the initial attempt fails. For more information see What is a dynamic value reference?.

Quickly view cluster details when you create a new cluster

April 20-26, 2021: Version 3.44

You will now see cluster information, including worker node and driver details, to the right of Create Cluster when you create a new all-purpose or job cluster.

MLflow sidebar reflects the most recent experiment

April 20-26, 2021: Version 3.44

The MLflow Experiment Runs sidebar now displays runs from the experiment that the notebook most recently logged to. Previously the sidebar showed runs only from the notebook experiment.

For details, see Track ML and deep learning training runs.

Change to default channel for conda.yaml files in MLflow

April 20-26, 2021: Version 3.44

When you save a model to MLflow, you can specify a conda.yaml file specifying package dependencies. Previously, if you did not specify channels in the conda.yaml file, the model used the defaults channel. This has changed. If you do not specify a channel, the new default channel is conda-forge.

Streamlined authentication from Tableau Online to Azure Databricks using Microsoft Entra ID

April 15, 2021

You can now authenticate from Tableau Online using Microsoft Entra ID authentication in addition to personal access tokens. For details, see Connect Tableau and Azure Databricks.

Databricks Runtime 8.2 (Beta)

April 8, 2021

Databricks Runtime 8.2 and Databricks Runtime 8.2 ML are now available as Beta releases.

For information, see the full release notes at Databricks Runtime 8.2 (EoS) and Databricks Runtime 8.2 for ML (EoS).

New extended infrastructure requires update to custom routes and egress firewalls

April 5, 2021

On May 31, Databricks will add extended infrastructure to most Azure regions to help support minimum-impact maintenance and upgrades. If you use custom routes or egress firewalls for your Azure Databricks VNet, you must add the regional extended infrastructure IP ranges to your egress firewalls or custom routes by May 31. For the list of new IP ranges, see Configure user-defined routes with IP addresses.

User and group limits

April 5-12, 2021: Version 3.43

Each Azure Databricks workspace is now limited to 10,000 users and 5,000 groups.

Easier monitoring of job run status

April 5-12, 2021: Version 3.43

The job run details page now automatically refreshes every 5 seconds to make it easier to monitor the progress of your jobs.

Global init scripts no longer run on model serving clusters

April 5-12, 2021: Version 3.43

Global init scripts are run on every cluster in a workspace and can be used to enforce consistent cluster configurations. This configuration is typically not optimal for model serving clusters, so global init scripts are no longer run on model serving clusters.

If you need to run init scripts on model serving clusters, contact your Azure Databricks account team.

Receive email notification about activity in Model Registry

April 5-12, 2021: Version 3.43

You can now configure Model Registry to send an email notification of activity on a registered model or model version. Creators of model versions and registered models automatically receive notifications about activity on their models. Users who interact with model versions (for example, by adding a comment) also automatically receive notifications. To learn how to manage these notifications, see Control notification preferences.

Databricks Runtime 6.4 series support ends

April 1, 2021

Support for Databricks Runtime 6.4, Databricks Runtime 6.4 for Machine Learning, and Databricks Runtime 6.4 for Genomics ended on April 1. See Databricks support lifecycles.

Databricks Runtime 6.4 Extended Support will be supported through the end of 2021. For more information, see Databricks Runtime 6.4 Extended Support (EoS).