Databricks Runtime 11.3 LTS

The following release notes provide information about Databricks Runtime 11.3 LTS, powered by Apache Spark 3.3.0. Databricks released this version in October 2022.

Behavioral changes

[Breaking change] New Python version requires updating Databricks Connect V1 Python clients

To apply required security patches, the Python version in Databricks Runtime 11.3 LTS is upgraded from 3.9.5 to 3.9.19. Because these changes might cause errors in clients that use specific PySpark functions, any clients that use Databricks Connect V1 for Python with Databricks Runtime 11.3 LTS must be updated to Python 3.9.7 or later.

New features and improvements

• Structured Streaming trigger once is deprecated
• Change source path for Auto Loader
• Databricks Kinesis connector now supports reading from Kinesis Data streams in EFO mode
• New H3 geospatial functions and added Photon support for all H3 functions
• New features for Predictive I/O
• Increasing initial partitions to scan for selective queries
• New AQE plan versions visualization
• New asynchronous progress tracking and log purging modes
• Structured Streaming on Unity Catalog now supports display()
• Pipeline events are now logged in JSON format
• Arbitrary Stateful Processing in Structured Streaming with Python
• Date inference in CSV files
• Clone support for Apache Parquet and Apache Iceberg tables (Public Preview)
• Use SQL to specify schema- and catalog-level storage locations for Unity Catalog managed tables

Structured Streaming trigger once is deprecated

The Trigger.Once setting has been deprecated. Databricks recommends you use Trigger.AvailableNow. See Configure Structured Streaming trigger intervals.

Change source path for Auto Loader

You can now change the directory input path for Auto Loader configured with directory listing mode without having to choose a new checkpoint directory. See Change source path for Auto Loader.

Databricks Kinesis connector now supports reading from Kinesis Data streams in EFO mode

You can now use the Databricks Kinesis structured streaming source in Databricks Runtime 11.3 LTS to run queries that read from Kinesis Data streams in enhanced fan-out mode. This allows dedicated throughput per shard, per consumer and record delivery in push mode.

New H3 geospatial functions and added Photon support for all H3 functions

Introducing 4 new H3 functions, h3_maxchild, h3_minchild, h3_pointash3, and h3_pointash3string. These functions are available in SQL, Scala, and Python. All H3 expressions are now supported in Photon. See H3 geospatial functions.

New features for Predictive I/O

Photon supports range mode for running frames, using RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. Photon also supports range mode for growing frames, using RANGE BETWEEN UNBOUNDED PRECEDING AND offset_stop { PRECEDING | FOLLOWING }.

Increasing initial partitions to scan for selective queries

The value of initial partitions to scan has been increased to 10 for selective query with take/tail/limit in Photon-enabled clusters and LIMIT in Databricks SQL. With 10 partitions, you can avoid the overhead of launching multiple small jobs and a slow scale-up. You can also configure this through spark.sql.limit.selectiveInitialNumPartitions.

New AQE plan versions visualization

Introducing AQE plan versions that allows you to visualize your runtime plan updates from adaptive query execution (AQE).

New asynchronous progress tracking and log purging modes

Introducing Structured Streaming modes called asynchronous progress tracking and asynchronous log purging. Asynchronous log purging mode lowers the latency of streaming queries by removing logs used for progress tracking in the background.

Structured Streaming on Unity Catalog now supports display()

You can now use display() when you use Structured Streaming to work with tables registered in Unity Catalog.

Pipeline events are now logged in JSON format

Azure Databricks now writes pipeline events to the driver log in JSON format. While each event will be JSON-parseable, large events may not contain all fields, or the fields may be truncated. Each event is logged in a single line with the prefix Event received: . The following is an example event.

Event received: {"id":"some-event-id","origin":{"pipeline_id":"some-pipeline-id","cluster_id":"some-cluster id"},"message":"simple [truncated] message","level":"WARN"}

Arbitrary Stateful Processing in Structured Streaming with Python

Introducing the applyInPandasWithState function that can be used to perform arbitrary stateful processing in PySpark. This is equivalent to the flatMapGroupsWithState function in the Java API.

Date inference in CSV files

Introducing improved inference of date type columns in CSV files. When the date format is consistent across the records for a column, those columns can be inferred as DateType. You can also have a combination of date formats across different columns. Azure Databricks can automatically infer the date format for each column. Date columns in CSV files prior to Databricks Runtime 11.3 LTS are left as StringType.

Clone support for Apache Parquet and Apache Iceberg tables (Public Preview)

Clone can now be used to create and incrementally update Delta tables that mirror Apache Parquet and Apache Iceberg tables. You can update your source Parquet table and incrementally apply the changes to their cloned Delta table with the clone command. See Incrementally clone Parquet and Iceberg tables to Delta Lake.

Use SQL to specify schema- and catalog-level storage locations for Unity Catalog managed tables

You can now use the MANAGED LOCATION SQL command to specify a cloud storage location for managed tables at the catalog and schema levels. See CREATE CATALOG and CREATE SCHEMA.

Behavior changes

Databricks Connect 11.3.2

Databricks Connect client update 11.3.2 is now supported. See Databricks Connect and Databricks Connect release notes.

Updated Azure Databricks Snowflake connector

The Azure Databricks Snowflake connector has been updated to the latest version of code from the open-source repository, Snowflake Data Source for Apache Spark. It is now fully compatible with Databricks Runtime 11.3 LTS, including predicate pushdown and internal query plan pushdown while maintaining all of the features of the open-source version.

Hadoop cache for S3A is now disabled

The Hadoop cache (FileSystem Apache Hadoop Main 3.3.4 API) for S3A is now disabled. This is to align with other cloud storage connectors. For workloads that rely on file system caching, make sure that newly created file systems are supplied with the correct Hadoop configurations, including credential providers.

Delta Lake stats collection schema now matches column order in table schema definition

This change addresses a bug in the Delta Lake protocol where stats were not collected for columns due to a mismatch in DataFrame and table column ordering. In some cases, you might encounter write performance degradation due to stats collection on previously untracked fields. See Data skipping for Delta Lake.

applyInPandasWithState throws an error if the query has a shuffle after the operator

The operator applyInPandasWithState throws an error if the query has shuffle after the operator. This happens when either the user adds shuffle after the operation, or the optimizer or sink adds shuffle implicitly.

Library upgrades

• Upgraded Python libraries:
  • distlib from 0.3.5 to 0.3.6
• Upgraded R libraries:
  • broom from 1.0.0 to 1.0.1
  • callr from 3.7.1 to 3.7.2
  • dplyr from 1.0.9 to 1.0.10
  • dtplyr from 1.2.1 to 1.2.2
  • forcats from 0.5.1 to 0.5.2
  • future from 1.27.0 to 1.28.0
  • future.apply from 1.9.0 to 1.9.1
  • gert from 1.7.0 to 1.8.0
  • globals from 0.16.0 to 0.16.1
  • gtable from 0.3.0 to 0.3.1
  • haven from 2.5.0 to 2.5.1
  • hms from 1.1.1 to 1.1.2
  • httr from 1.4.3 to 1.4.4
  • knitr from 1.39 to 1.40
  • modelr from 0.1.8 to 0.1.9
  • pillar from 1.8.0 to 1.8.1
  • progressr from 0.10.1 to 0.11.0
  • readxl from 1.4.0 to 1.4.1
  • reprex from 2.0.1 to 2.0.2
  • rlang from 1.0.4 to 1.0.5
  • rmarkdown from 2.14 to 2.16
  • RSQLite from 2.2.15 to 2.2.16
  • rstudioapi from 0.13 to 0.14
  • rversions from 2.1.1 to 2.1.2
  • rvest from 1.0.2 to 1.0.3
  • scales from 1.2.0 to 1.2.1
  • sparklyr from 1.7.7 to 1.7.8
  • stringr from 1.4.0 to 1.4.1
  • survival from 3.2-13 to 3.4-0
  • tinytex from 0.40 to 0.41
  • viridisLite from 0.4.0 to 0.4.1
• Upgraded Java libraries:
  • com.fasterxml.jackson.core.jackson-annotations from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.core.jackson-core from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.core.jackson-databind from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.dataformat.jackson-dataformat-cbor from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.datatype.jackson-datatype-joda from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.datatype.jackson-datatype-jsr310 from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.module.jackson-module-paranamer from 2.13.3 to 2.13.4
  • com.fasterxml.jackson.module.jackson-module-scala_2.12 from 2.13.3 to 2.13.4
  • org.apache.hadoop.hadoop-client-api from 3.3.2-databricks to 3.3.4-databricks
  • org.apache.hadoop.hadoop-client-runtime from 3.3.2 to 3.3.4
  • org.apache.orc.orc-core from 1.7.5 to 1.7.6
  • org.apache.orc.orc-mapreduce from 1.7.5 to 1.7.6
  • org.apache.orc.orc-shims from 1.7.5 to 1.7.6
  • org.apache.parquet.parquet-column from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
  • org.apache.parquet.parquet-common from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
  • org.apache.parquet.parquet-encoding from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
  • org.apache.parquet.parquet-format-structures from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
  • org.apache.parquet.parquet-hadoop from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
  • org.apache.parquet.parquet-jackson from 1.12.0-databricks-0004 to 1.12.0-databricks-0007
  • org.glassfish.jersey.containers.jersey-container-servlet from 2.34 to 2.36
  • org.glassfish.jersey.containers.jersey-container-servlet-core from 2.34 to 2.36
  • org.glassfish.jersey.core.jersey-client from 2.34 to 2.36
  • org.glassfish.jersey.core.jersey-common from 2.34 to 2.36
  • org.glassfish.jersey.core.jersey-server from 2.34 to 2.36
  • org.glassfish.jersey.inject.jersey-hk2 from 2.34 to 2.36

Apache Spark

Databricks Runtime 11.3 LTS includes Apache Spark 3.3.0. This release includes all Spark fixes and improvements included in Databricks Runtime 11.2 (EoS), as well as the following additional bug fixes and improvements made to Spark:

• [SPARK-39957] [WARMFIX][SC-111425][CORE] Delay onDisconnected to enable Driver receives ExecutorExitCode
• [SPARK-39955] [WARMFIX][SC-111424][CORE] Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages
• [SPARK-40474] [SC-106248][Cherry-Pick] Correct CSV schema inference behavior for datetime columns and introduce auto detection for Date fields
• [SPARK-40535] [SC-111243][SQL] Fix bug the buffer of AggregatingAccumulator will not be created if the input rows is empty
• [SPARK-40434] [SC-111125][SC-111144][SC-111138][SPARK-40435][11.3][SS][PYTHON] Implement applyInPandasWithState in PySpark
• [SPARK-40460] [SC-110832][SS] Fix streaming metrics when selecting _metadata
• [SPARK-40324] [SC-109943][SQL] Provide a query context of ParseException
• [SPARK-40466] [SC-110899][SS] Improve the error message when DSv2 is disabled while DSv1 is not available
• [SPARK-40456] [SC-110848][SQL] PartitionIterator.hasNext should be cheap to call repeatedly
• [SPARK-40169] [SC-110772][SQL] Don’t pushdown Parquet filters with no reference to data schema
• [SPARK-40467] [SC-110759][SS] Split FlatMapGroupsWithState down to multiple test suites
• [SPARK-40468] [SC-110813][SQL] Fix column pruning in CSV when _corrupt_record is selected
• [SPARK-40291] [SC-110085][SQL] Improve the message for column not in group by clause error
• [SPARK-40398] [SC-110762][CORE][SQL] Use Loop instead of Arrays.stream api
• [SPARK-40433] [SC-110684][SS][PYTHON] Add toJVMRow in PythonSQLUtils to convert pickled PySpark Row to JVM Row
• [SPARK-40414] [SC-110568][SQL][PYTHON] More generic type on PythonArrowInput and PythonArrowOutput
• [SPARK-40352] [SC-109945][SQL] Add function aliases: len, datepart, dateadd, date_diff and curdate
• [SPARK-40470] [SC-110761][SQL] Handle GetArrayStructFields and GetMapValue in “arrays_zip” function
• [SPARK-40387] [SC-110685][SQL] Improve the implementation of Spark Decimal
• [SPARK-40429] [SC-110675][SQL] Only set KeyGroupedPartitioning when the referenced column is in the output
• [SPARK-40432] [SC-110716][SS][PYTHON] Introduce GroupStateImpl and GroupStateTimeout in PySpark
• [SPARK-39915] [SC-110496][SQL] Ensure the output partitioning is user-specified in AQE
• [SPARK-29260] [SQL] Support ALTER DATABASE SET LOCATION if HMS supports
• [SPARK-40185] [SC-110056][SQL] Remove column suggestion when the candidate list is empty
• [SPARK-40362] [SC-110401][SQL] Fix BinaryComparison canonicalization
• [SPARK-40411] [SC-110381][SS] Refactor FlatMapGroupsWithStateExec to have a parent trait
• [SPARK-40293] [SC-110084][SQL] Make the V2 table error message more meaningful
• [SPARK-38734] [SC-110383][SQL] Remove the error class INDEX_OUT_OF_BOUNDS
• [SPARK-40292] [SC-110300][SQL] Fix column names in “arrays_zip” function when arrays are referenced from nested structs
• [SPARK-40276] [SC-109674][CORE] Reduce the result size of RDD.takeOrdered
• [SPARK-40197] [SC-109176][SQL] Replace query plan with context for MULTI_VALUE_SUBQUERY_ERROR
• [SPARK-40300] [SC-109942][SQL] Migrate onto the DATATYPE_MISMATCH error class
• [SPARK-40149] [SC-110055][SQL] Propagate metadata columns through Project
• [SPARK-40280] [SC-110146][SQL] Add support for parquet push down for annotated int and long
• [SPARK-40220] [SC-110143][SC-109175][SQL] Don’t output the empty map of error message parameters
• [SPARK-40295] [SC-110070][SQL] Allow v2 functions with literal args in write distribution/ordering
• [SPARK-40156] [SC-109264][SQL] url_decode() should the return an error class
• [SPARK-39195] [SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
• [SPARK-40260] [SC-109424][SQL] Use error classes in the compilation errors of GROUP BY a position
• [SPARK-40205] [SC-110144][SC-109082][SQL] Provide a query context of ELEMENT_AT_BY_INDEX_ZERO
• [SPARK-40112] [SC-109676][SQL] Improve the TO_BINARY() function
• [SPARK-40209] [SC-109081][SQL] Don’t change the interval value of Decimal in changePrecision() on errors
• [SPARK-40319] [SC-109873][SQL] Remove duplicated query execution error method for PARSE_DATETIME_BY_NEW_PARSER
• [SPARK-40222] [SC-109209][SQL] Numeric try_add/try_divide/try_subtract/try_multiply should throw error from their children
• [SPARK-40183] [SC-108907][SQL] Use error class NUMERIC_VALUE_OUT_OF_RANGE for overflow in decimal conversion
• [SPARK-40180] [SC-109069][SQL] Format error messages by spark-sql
• [SPARK-40153] [SC-109165][SQL] Unify resolve functions and table-valued functions
• [SPARK-40308] [SC-109880][SQL] Allow non-foldable delimiter arguments to str_to_map function
• [SPARK-40219] [SC-110052][SC-109663][SQL] Resolved view logical plan should hold the schema to avoid redundant lookup
• [SPARK-40098] [SC-109939][SC-108693][SQL] Format error messages in the Thrift Server
• [SPARK-39917] [SC-109038][SQL] Use different error classes for numeric/interval arithmetic overflow
• [SPARK-40033] [SC-109875][SQL] Nested schema pruning support through element_at
• [SPARK-40194] [SC-109660][SQL] SPLIT function on empty regex should truncate trailing empty string.
• [SPARK-40228] [SC-109835][SQL] Do not simplify multiLike if child is not a cheap expression
• [SPARK-40039] [SC-109896][SC-109260][SS] Introducing a streaming checkpoint file manager based on Hadoop’s Abortable interface
• [SPARK-40285] [SC-109679][SQL] Simplify the roundTo[Numeric] for Spark Decimal
• [SPARK-39896] [SC-109658][SQL] UnwrapCastInBinaryComparison should work when the literal of In/InSet downcast failed
• [SPARK-40040] [SC-109662][SQL] Push local limit to both sides if join condition is empty
• [SPARK-40055] [SC-109075][SQL] listCatalogs should also return spark_catalog even when spark_catalog implementation is defaultSessionCatalog
• [SPARK-39915] [SC-109391][SQL] Dataset.repartition(N) may not create N partitions Non-AQE part
• [SPARK-40207] [SC-109401][SQL] Specify the column name when the data type is not supported by datasource
• [SPARK-40245] [SC-109295][SQL] Fix FileScan equality check when partition or data filter columns are not read
• [SPARK-40113] [SC-109405][SQL] Reactor ParquetScanBuilder DataSourceV2 interface implementations
• [SPARK-40211] [SC-109226][CORE][SQL] Allow customize initial partitions number in take() behavior
• [SPARK-40252] [SC-109379][SQL] Replace Stream.collect(Collectors.joining) with StringJoiner Api
• [SPARK-40247] [SC-109272][SQL] Fix BitSet equality check
• [SPARK-40067] [SQL] Use Table#name() instead of Scan#name() to populate the table name in the BatchScan node in SparkUI
• [SPARK-39966] [SQL] Use V2 Filter in SupportsDelete
• [SPARK-39607] [SC-109268][SQL][DSV2] Distribution and ordering support V2 function in writing
• [SPARK-40224] [SC-109271][SQL] Make ObjectHashAggregateExec release memory eagerly when fallback to sort-based
• [SPARK-40013] [SQL] DS V2 expressions should have the default toString
• [SPARK-40214] [SC-109079][PYTHON][SQL] add ‘get’ to functions
• [SPARK-40192] [SC-109089][SQL][ML] Remove redundant groupby
• [SPARK-40146] [SC-108694][SQL] Simply the codegen of getting map value
• [SPARK-40109] [SQL] New SQL function: get()
• [SPARK-39929] [SQL] DS V2 supports push down string functions(non ANSI)
• [SPARK-39819] [SQL] DS V2 aggregate push down can work with Top N or Paging (Sort with expressions)
• [SPARK-40213] [SC-109077][SQL] Support ASCII value conversion for Latin-1 characters
• [SPARK-39887] [SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
• [SPARK-39764] [SQL] Make PhysicalOperation the same as ScanOperation
• [SPARK-39964] [SQL] DS V2 pushdown should unify the translate path
• [SPARK-39528] [SQL] Use V2 Filter in SupportsRuntimeFiltering
• [SPARK-40066] [SQL] ANSI mode: always return null on invalid access to map column
• [SPARK-39912] [SPARK-39828][SQL] Refine CatalogImpl
• [SPARK-39833] [SC-108736][SQL] Disable Parquet column index in DSv1 to fix a correctness issue in the case of overlapping partition and data columns
• [SPARK-39880] [SQL] V2 SHOW FUNCTIONS command should print qualified function name like v1
• [SPARK-39767] [SQL] Remove UnresolvedDBObjectName and add UnresolvedIdentifier
• [SPARK-40163] [SC-108740][SQL] feat: SparkSession.config(Map)
• [SPARK-40136] [SQL] Fix the fragment of SQL query contexts
• [SPARK-40107] [SC-108689][SQL] Pull out empty2null conversion from FileFormatWriter
• [SPARK-40121] [PYTHON][SQL] Initialize projection used for Python UDF
• [SPARK-40128] [SQL] Make the VectorizedColumnReader recognize DELTA_LENGTH_BYTE_ARRAY as a standalone column encoding
• [SPARK-40132] [ML] Restore rawPredictionCol to MultilayerPerceptronClassifier.setParams
• [SPARK-40050] [SC-108696][SQL] Enhance EliminateSorts to support removing sorts via LocalLimit
• [SPARK-39629] [SQL] Support v2 SHOW FUNCTIONS
• [SPARK-39925] [SC-108734][SQL] Add array_sort(column, comparator) overload to DataFrame operations
• [SPARK-40117] [PYTHON][SQL] Convert condition to java in DataFrameWriterV2.overwrite
• [SPARK-40105] [SQL] Improve repartition in ReplaceCTERefWithRepartition
• [SPARK-39503] [SQL] Add session catalog name for v1 database table and function
• [SPARK-39889] [SQL] Use different error classes for numeric/interval divided by 0
• [SPARK-39741] [SQL] Support url encode/decode as built-in function and tidy up url-related functions
• [SPARK-40102] [SQL] Use SparkException instead of IllegalStateException in SparkPlan
• [SPARK-40014] [SQL] Support cast of decimals to ANSI intervals
• [SPARK-39776] [SQL][FOLLOW] Update UT of PlanStabilitySuite in ANSI mode
• [SPARK-39963] [SQL] Simplify SimplifyCasts.isWiderCast

Maintenance updates

See Databricks Runtime 11.3 maintenance updates.

System environment

• Operating System: Ubuntu 20.04.5 LTS
• Java: Zulu 8.56.0.21-CA-linux64
• Scala: 2.12.14
• Python: 3.9.19
• R: 4.1.3
• Delta Lake: 2.1.0

Installed Python libraries

Installed R libraries

R libraries are installed from the Microsoft CRAN snapshot on 2022-09-08.

Installed Java and Scala libraries (Scala 2.12 cluster version)