Fabric Runtime 1.1 (EOSA)
Microsoft Fabric Runtime is an Azure-integrated platform based on Apache Spark that enables the execution and management of the Data Engineering and Data Science experiences in Fabric. This document covers the Fabric Runtime 1.1 components and versions.
Warning
The end of support date for Runtime 1.1 has been announced as July 12, 2024. Runtime 1.1, based on Apache Spark 3.3, will be deprecated and disabled as of March 31, 2025. Upgrade your workspace and environments to Runtime 1.2 or Runtime 1.3. For the complete lifecycle and support policies of Apache Spark runtimes in Fabric, refer to Lifecycle of Apache Spark runtimes in Fabric.
Microsoft Fabric Runtime 1.1 is one of the runtimes offered within the Microsoft Fabric platform. The Runtime 1.1 major components are:
- Apache Spark 3.3
- Operating System: Ubuntu 18.04
- Java: 1.8.0_282
- Scala: 2.12.15
- Python: 3.10
- Delta Lake: 2.2
- R: 4.2.2
Tip
Always use the most recent, GA runtime version for your production workload, which currently is Runtime 1.3.
Microsoft Fabric Runtime 1.1 comes with a collection of default level packages, including a full Anaconda installation and commonly used libraries for Java/Scala, Python, and R. These libraries are automatically included when using notebooks or jobs in the Microsoft Fabric platform. Refer to the documentation for a complete list of libraries.
Microsoft Fabric periodically releases maintenance updates for Runtime 1.1, delivering bug fixes, performance enhancements, and security patches. Ensuring you stay up to date with these updates guarantees optimal performance and reliability for your data processing tasks. If you are currently using Runtime 1.1, you can upgrade to Runtime 1.3 or to Runtime 1.2 by navigating to Workspace Settings > Data Engineering / Science > Spark Settings > Environment.
New features and improvements - Apache Spark 3.3.1
Read the full version of the release notes for a specific Apache Spark version by visiting both Spark 3.3.0 and Spark 3.3.1.
New features and improvements - Delta Lake 2.2
Check the source and full release notes at Delta Lake 2.2.0.
Default-level packages for Java/Scala
For a list of all the default level packages for Java, Scala, Python and their respective versions see the release notes.
Migration between different Apache Spark versions
Migrating your workloads to Fabric Runtime 1.1 (Apache Spark 3.3) from an older version of Apache Spark involves a series of steps to ensure a smooth migration. This guide outlines the necessary steps to help you migrate efficiently and effectively.
Review Fabric Runtime 1.1 release notes, including checking the components and default-level packages included into the runtime, to understand the new features and improvements.
Check compatibility of your current setup and all related libraries, including dependencies and integrations. Review the migration guides to identify potential breaking changes:
- Review the Spark Core migration guide.
- Review the SQL, Datasets and DataFrame migration guide.
- If your solution is Apache Spark Structure Streaming related, review the Structured Streaming migration guide.
- If you use PySpark, review the Pyspark migration guide.
- If you migrate code from Koalas to PySpark, review the Koalas to pandas API on Spark migration guide.
Move your workloads to Fabric and ensure that you have backups of your data and configuration files in case you need to revert to the previous version.
Update any dependencies that the new version of Apache Spark or other Fabric Runtime 1.1 related components might impact, including third-party libraries or connectors. Make sure to test the updated dependencies in a staging environment before deploying to production.
Update the Apache Spark configuration on your workload, including updating configuration settings, adjusting memory allocations, and modifying any deprecated configurations.
Modify your Apache Spark applications (notebooks and Apache Spark job definitions) to use the new APIs and features introduced in Fabric Runtime 1.1 and Apache Spark 3.3. You might need to update your code to accommodate any deprecated or removed APIs, and refactor your applications to take advantage of performance improvements and new functionalities.
Thoroughly test your updated applications in a staging environment to ensure compatibility and stability with Apache Spark 3.3. Perform performance testing, functional testing, and regression testing to identify and resolve any issues that might arise during the migration process.
After validating your applications in a staging environment, deploy the updated applications to your production environment. Monitor the performance and stability of your applications after the migration to identify any issues that need to be addressed.
Update your internal documentation and training materials to reflect the changes introduced in Fabric Runtime 1.1. Ensure that your team members are familiar with the new features and improvements to maximize the benefits of the migration.