August 2020

  • Article

These features and Azure Databricks platform improvements were released in August 2020.

Note

Releases are staged. Your Azure Databricks account may not be updated until up to a week after the initial release date.

Important

Version 3.26 has been released to customers in the Canada Central and Central India regions only. All other regions will get the 3.26 features at the same time that 3.27 is released.

Token Management API is GA and admins can use the Admin Console to grant and revoke user access to tokens

August 26 - September 1, 2020: Version 3.27

Token management is now generally available. Azure Databricks administrators can use the Token Management API and the Admin Console to manage their users’ Azure Databricks personal access tokens. As an admin, you can:

  • Monitor and revoke users’ personal access tokens.
  • Control the lifetime of future tokens in your workspace.
  • Control which users can create and use tokens via the Permissions API or in the Admin Console.

In the transition from Public Preview to GA, the Token Management API parameter created_by was changed to created_by_id, and a new parameter, created_by_username was added.

For more information, see Monitor and revoke personal access tokens.

Message size limits for Shiny apps increased

August 26 - September 1, 2020: Version 3.27

The maximum application size for Shiny apps has been increased from 10 MB to 20 MB. If your application’s total size exceeds this limit, review the recommendations in the Shiny FAQ.

Improved instructions for setting up a cluster in local mode

August 26 - September 1, 2020: Version 3.27

In the cluster UI:

  • If you create a cluster with 0 workers, a tool tip appears recommending that you use local mode and showing the associated configuration setting (spark.master local[*]).
  • You can no longer set spark.master local[*] for a cluster, unless the cluster has 0 workers.

View version of notebook associated with a run

August 26 - September 1, 2020: Version 3.27

From the Experiments sidebar, you can now display the version of a notebook associated with a run. For details, see View notebook experiment.

Databricks Runtime 7.2 GA

August 20, 2020

Databricks Runtime 7.2 brings many additional features and improvements over Databricks Runtime 7.1, including:

  • Auto Loader is generally available: Auto Loader is an efficient method for incrementally ingesting a large number of files into Delta Lake. It is now GA and adds the following features:
    • Directory listing mode option: Auto Loader adds a new directory listing mode, in addition to the existing file notification mode, for determining when there are new files.
    • Cloud resource management API: You can now use our Scala API to manage cloud resources created by Auto Loader. You can list notification services and tear down specific notification services using this API.
    • Rate limiting option: You can now use the cloudFiles.maxBytesPerTrigger option to limit the amount of data processed in each microbatch.
    • Option validation: Auto Loader now validates the options you provide.validation will fail. To skip option validation, set cloudFiles.validateOptions to false.
  • Efficiently copy a Delta table with clone.
  • Improvements:
    • Snowflake connector has been upgraded to version 2.8.1, which includes Spark 3.0 support.
    • Credential passthrough improvements
    • TensorBoard improvements
    • Upgraded Python and R libraries

For details, see the complete Databricks Runtime 7.2 (EoS) release notes.

Databricks Runtime 7.2 ML GA

August 20, 2020

Databricks Runtime 7.2 for Machine Learning is built on top of Databricks Runtime 7.2 and brings new and improved Python and system libraries. For details, see the complete Databricks Runtime 7.2 (EoS) release notes.

Databricks Runtime 7.2 Genomics GA

August 20, 2020

Databricks Runtime 7.2 for Genomics is built on top of Databricks Runtime 7.2 and significantly speeds up the conversion of literal numpy 1D and 2D float-typed ndarrays to Java arrays. The Glow genome-wide association study documentation reflects the usage.

Permissions API (Public Preview)

August 18, 2020

Databricks is pleased to announce the public preview of the Permissions API, which lets you manage permissions for:

  • Tokens
  • Clusters
  • Pools
  • Jobs
  • Notebooks
  • Folders (directories)
  • MLflow registered models

For more information, see Permissions API.

Databricks Connect 7.1 (GA)

August 12, 2020

Databricks Connect now supports Databricks Runtime 7.1.

In Databricks Runtime 7.1, Databricks recommends that you always use the most recent version of Databricks Connect.

Repeatable installation order for cluster libraries

August 12-25, 2020: Version 3.26

On a cluster running Databricks Runtime 7.2 or above, Azure Databricks now processes all cluster libraries in the order that they were installed.

Create model from MLflow registered models page (Public Preview)

August 12-25, 2020: Version 3.26

You can now create a new model from the MLflow registered models page. For details, see Create a new registered model and assign a logged model to it.

Databricks Container Services supports GPU images

August 12-25, 2020: Version 3.26

You can now use Databricks Container Services on clusters with GPUs to create portable deep learning environments with customized libraries.

For details, see Databricks Container Services on GPU compute.