What's new archive for Azure Data Factory

Azure Data Factory is improved on an ongoing basis. To stay up to date with the most recent developments, refer to the current What's New page, which provides you with information about:

  • The latest releases.
  • Known issues.
  • Bug fixes.
  • Deprecated functionality.
  • Plans for changes.

This archive page retains updates from older months.

Check out our What's New video archive for all of our monthly updates.

September 2023

Pipelines

Added support for metadata driven pipelines for dynamic full and incremental processing in Azure SQL Learn more

August 2023

Change Data Capture

  • Azure Synapse Analytics target availability in top-level CDC resource Learn more
  • Snowflake connector in Mapping Data Flows support for Change Data Capture in public preview Learn more

Data flow

  • Integer type available for pipeline variables Learn more
  • Snowflake CDC source connector available in top-level CDC resource Learn more
  • Native UI support of parameterization for more linked services Learn more

Data movement

  • Managed private endpoints support for Application Gateway and MySQL Flexible Server Learn more
  • Managed virtual network time-to-live (TTL) general availability Learn more

Integration runtime

Self-hosted integration runtime now supports self-contained interactive authoring (Preview) Learn more

July 2023

Change Data Capture

Top-level CDC resource now supports schema evolution. Learn more

Data flow

Merge schema option in delta sink now supports schema evolution in Mapping Data Flows. Learn more

Data movement

  • Comment Out Part of Pipeline with Deactivation. Learn more
  • Pipeline return value is now generally available. Learn more

Developer productivity

Documentation search now included in the Azure Data Factory search toolbar. Learn more

June 2023

Continuous integration and continuous deployment

npm package now supports pre-downloaded bundle for building ARM templates. If your firewall setting is blocking direct download for your npm package, you can now pre-load the package upfront, and let npm package consume local version instead. This is a super boost for your CI/CD pipeline in a firewalled environment.

Region expansion

Azure Data Factory is now available in Sweden Central. You can co-locate your ETL workflow in this new region if you are utilizing the region for storing and managing your modern data warehouse. Learn more

Data movement

Securing outbound traffic with Azure Data Factory's outbound network rules is now supported. Learn more

Connectors

The Amazon S3 connector is now supported as a sink destination using Mapping Data Flows. Learn more

Data flow

We introduce optional Source settings for DelimitedText and JSON sources in top-level CDC resource. The top-level CDC resource in data factory now supports optional source configurations for Delimited and JSON sources. You can now select the column/row delimiters for delimited sources and set the document type for JSON sources. Learn more

May 2023

Data Factory in Microsoft Fabric

Data factory in Microsoft Fabric provides cloud-scale data movement and data transformation services that allow you to solve the most complex data factory and ETL scenarios. It's intended to make your data factory experience easy to use, powerful, and truly enterprise-grade.

April 2023

Data flow

Easily unroll multiple arrays in ADF data flows. ADF updated the Flatten transformation that now makes it super easy to unroll multiple arrays from a single Flatten transformation step. Learn more

Continuous integration and continuous deployment

You can customize the commit message in Git mode now. Type in a detailed description about the changes you make, and we will save it to Git repository.

Connectors

The Azure Blob Storage connector now supports anonymous authentication. Learn more

March 2023

Connectors

Azure Data Lake Storage Gen2 connector now supports shared access signature authentication. Learn more

February 2023

Data movement

  • Anonymous authentication type supported for Azure Blob storage Learn more
  • Updated SAP template to easily move SAP data to ADLSGen2 in Delta format Learn more

Orchestration

  • Set pipeline output value (Public preview) Learn more
  • Workflow Orchestration Manager (Public preview) Learn more

Developer productivity

Dark theme support added (Public preview) Learn more

January 2023

Data flow

  • Flowlets now support schema drift Learn more
  • SQL CDC incremental extract now supports numeric columns Learn more

Data movement

New top-level CDC resource - native CDC configuration in 3 simple steps Learn more

Orchestration

Orchestrate Synapse notebooks and Synapse spark job definitions natively Learn more

Region expansion

  • Continued region expansion - China North 3 now supported Learn more
  • Continued region expansion - Switzerland West now supported Learn more

SQL Server Integration Services (SSIS)

Express virtual network injection for SSIS now generally available Learn more

Developer productivity

  • Added support for favoriting resources on home page
  • Save column width for pipeline monitoring
  • Monitor filter updates for faster searches
  • Directly launch Pipeline Template Gallery through Azure portal

December 2022

Data flow

SQL change data capture (CDC) incremental extract - supports numeric columns in mapping dataflow Learn more

Data movement

Express virtual network injection for SSIS in Azure Data Factory is generally available Learn more

Region expansion

Continued region expansion - Azure Data Factory is now available in China North 3 Learn more

November 2022

Data flow

  • Incremental only is available in SAP CDC - get changes only from SAP system without initial full load Learn more
  • Source partitions in initial full data load of SAP CDC to improve performance Learn more
  • A new pipeline template - Load multiple objects with big amounts from SAP via SAP CDC Learn more

Data Movement

  • Support to Azure Databricks through private link from a Data Factory managed virtual network Learn more

User Interface

2 Pipeline designer enhancements added to ADF Studio preview experience

  • Dynamic content flyout - make it easier to set dynamic content in your pipeline activities without using the expression builder Learn more
  • Error message relocation to status column - make it easier for you to view errors when you see a Failed pipeline run Learn more

Continuous integration and continuous deployment

In auto publish config, disable publish button is available to void overwriting the last automated publish deployment Learn more

October 2022

Video summary

Data flow

  • Export up to 1000 rows from data flow preview Learn more
  • SQL CDC in Mapping Data Flows now available (Public Preview) Learn more
  • Unlock advanced analytics with Microsoft 365 Mapping Data Flow Connector Learn more
  • SAP Change Data Capture (CDC) in now generally available Learn more

Developer productivity

  • Now accepting community contributions to Template Gallery Learn more
  • New design in Azure portal - easily discover how to launch ADF Studio Learn more
  • Learning Center now available in the Azure Data Factory studio Learn more
  • One-click to try Azure Data Factory Learn more

Orchestration

  • Granular billing view available for ADF - see detailed billing information by pipeline (Public Preview) Learn more
  • Script activity execution timeout now configurable Learn more

Region expansion

Continued region expansion - Qatar Central now supported Learn more

Continuous integration and continuous deployment

Exclude pipeline triggers that did not change in deployment now generally available Learn more

September 2022

Video summary

Data flow

  • Amazon S3 source connector added Learn more
  • Google Sheets REST-based connector added as Source (Preview) Learn more
  • Maximum column optimization in dataflow Learn more
  • SAP Change Data Capture capabilities in Mapping Data Flow (Preview) - Extract and transform data changes from SAP systems for a more efficient data refresh Learn more
  • Writing data to a lookup field via alternative keys supported in Dynamics 365/CRM connectors for mapping data flows Learn more

Data movement

Support to convert Oracle NUMBER type to corresponding integer in source Learn more

Monitoring

  • Additional monitoring improvements in Azure Data Factory Learn more
    • Monitoring loading improvements - pipeline re-run groupings data fetched only when expanded
    • Pagination added to pipeline activity runs view to show all activity records in pipeline run
    • Monitoring consumption improvement - loading icon added to know when consumption report is fully calculated
    • Additional sorting columns in monitoring - sorting added for Pipeline name, Run End, and Status
    • Time-zone settings now saved in monitoring
  • Gantt chart view now supported in IR monitoring Learn more

Orchestration

DELETE method in the Web activity now supports sending a body with HTTP request Learn more

User interface

  • Native UI support of parameterization added for 6 additional linked services - SAP ODP, ODBC, Microsoft Access, Informix, Snowflake, and DB2 Learn more
  • Pipeline designer enhancements added in Studio Preview experience - users can view workflow inside pipeline objects like For Each, If Then, etc. Learn more

August 2022

Video summary

Data flow

  • Appfigures connector added as Source (Preview) Learn more
  • Cast transformation added - visually convert data types Learn more
  • New UI for inline datasets - categories added to easily find data sources Learn more

Data movement

Service principal authentication type added for Azure Blob storage Learn more

Developer productivity

  • Default activity time-out changed from 7 days to 12 hours Learn more
  • New data factory creation experience - one click to have your factory ready within seconds Learn more
  • Expression builder UI update - categorical tabs added for easier use Learn more

Continuous integration and continuous delivery (CI/CD)

When CI/CD integrating ARM template, instead of turning off all triggers, it can exclude triggers that didn't change in deployment Learn more

July 2022

Video summary

Data flow

  • Asana connector added as source Learn more
  • Three new data transformation functions now supported Learn more
    • collectUnique() - Create a new collection of unique values in an array.
    • substringIndex() - Extract the substring before n occurrences of a delimiter.
    • topN() - Return the top n results after sorting your data.
  • Refetch from source available in Refresh for data source change scenarios Learn more
  • User defined functions (GA) - Create reusable and customized expressions to avoid building complex logic over and over Learn more Video
  • Easier configuration on data flow runtime - choose compute size among Small, Medium and Large to pre-configure all integration runtime settings Learn more

Continuous integration and continuous delivery (CI/CD)

Include Global parameters supported in ARM template. Learn more

Developer productivity

Be a part of Azure Data Factory studio preview features - Experience the latest Azure Data Factory capabilities and be the first to share your feedback Learn more

June 2022

Video summary

Service categoryService improvementsDetails
Data flowFuzzy join supported for data flowsFuzzy join is now supported in Join transformation of data flows with configurable similarity score on join conditions.
Learn more
Editing capabilities in source projectionEditing capabilities in source projection is available in Dataflow to make schemas modifications easily
Learn more
Assert error handlingAssert error handling are now supported in data flows for data quality and data validation
Learn more
Data MovementParameterization natively supported in additional 4 connectorsWe added native UI support of parameterization for the following linked services: Azure Data Lake Gen1, Azure PostgreSQL, Google AdWords and PostgreSQL.
Learn more
SAP Change Data Capture (CDC) capabilities in the new SAP ODP connector (Public Preview)SAP Change Data Capture (CDC) capabilities are now supported in the new SAP ODP connector.
Learn more
Integration RuntimeTime-To-Live in managed VNET (Public Preview)Time-To-Live can be set to the provisioned computes in managed VNET.
Learn more
Monitoring Rerun pipeline with new parametersYou can now rerun pipelines with new parameter values in Azure Data Factory.
Learn more
Orchestration‘turnOffAsync' property is available in Web activityWeb activity supports an async request-reply pattern that invokes HTTP GET on the Location field in the response header of an HTTP 202 Response. It helps web activity automatically poll the monitoring end-point till the job runs. ‘turnOffAsync' property is supported to disable this behavior in cases where polling isn't needed
Learn more

May 2022

Service categoryService improvementsDetails
Data flowUser Defined Functions for mapping data flowsAzure Data Factory introduces in public preview user defined functions and data flow libraries. A user defined function is a customized expression you can define to be able to reuse logic across multiple mapping data flows. User defined functions live in a collection called a data flow library to be able to easily group up common sets of customized functions.
Learn more

April 2022

Service categoryService improvementsDetails
Data flowData preview and debug improvements in mapping data flowsDebug sessions using the AutoResolve Azure integration runtime (IR) will now start up in under 10 seconds. There are new updates to the data preview panel in mapping data flows. Now you can sort the rows inside the data preview view by selecting column headers. You can move columns around interactively. You can also save the data preview results as a CSV by using Export CSV.
Learn more
Dataverse connector is available for mapping data flowsDataverse connector is available as source and sink for mapping data flows.
Learn more
Support for user database schemas for staging with the Azure Synapse Analytics and PostgreSQL connectors in data flow sinkData flow sink now supports using a user database schema for staging in both the Azure Synapse Analytics and PostgreSQL connectors.
Learn more
MonitoringMultiple updates to Data Factory monitoring experiencesNew updates to the monitoring experience in Data Factory include the ability to export results to a CSV, clear all filters, and open a run in a new tab. Column and result caching is also improved.
Learn more
User interfaceNew regional format supportChoosing your language and the regional format in settings will influence the format of how data such as dates and times appear in the Azure Data Factory Studio monitoring. For example, the time format in Monitoring will appear like "Apr 2, 2022, 3:40:29 pm" when choosing English as the regional format, and "2 Apr 2022, 15:40:29" when choosing French as regional format. These settings affect only the Azure Data Factory Studio user interface and do not change/ modify your actual data and time zone.

March 2022

Video summary

Service categoryService improvementsDetails
Data flowScriptLines and parameterized linked service support added mapping data flowsIt's now easy to detect changes to your data flow script in Git with ScriptLines in your data flow JSON definition. Parameterized linked services can now also be used inside your data flows for flexible generic connection patterns.
Learn more
Flowlets general availability (GA)Flowlets is now generally available to create reusable portions of data flow logic that you can share in other pipelines as inline transformations. Flowlets enable extract-transform-and-load (ETL) jobs to be composed of custom or common logic components.
Learn more
Change Feed connectors are available in five data flow source transformationsChange Feed connectors are available in data flow source transformations for Azure Cosmos DB, Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, and the common data model (CDM).
Learn more
Data preview and debug improvements in mapping data flowsNew features were added to data preview and the debug experience in mapping data flows.
Learn more
SFTP connector for mapping data flowSFTP connector is available for mapping data flow as both source and sink.
Learn more
Data movementSupport Always Encrypted for SQL-related connectors in Lookup activity under Managed virtual networkAlways Encrypted is supported for SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Synapse Analytics in the Lookup activity under managed virtual network.
Learn more
Integration runtimeNew UI layout in Azure IR creation and edit pageThe UI layout of the IR creation and edit page now uses tab style for Settings, Virtual network, and Data flow runtime.
Learn more
OrchestrationTransform data by using the Script activityYou can use a Script activity to invoke a SQL script in SQL Database, Azure Synapse Analytics, SQL Server, Oracle, or Snowflake.
Learn more
Web activity timeout improvementYou can configure response timeout in a Web activity to prevent it from timing out if the response period is more than one minute, especially in the case of synchronous APIs.
Learn more

February 2022

Video summary

Service categoryService improvementsDetails
Data flowParameterized linked services supported in mapping data flowsYou can now use your parameterized linked services in mapping data flows to make your data flow pipelines generic and flexible.
Learn more
SQL Database incremental source extract available in data flow (public preview)A new option has been added on mapping data flow SQL Database sources called Enable incremental extract (preview). Now you can automatically pull only the rows that have changed on your SQL Database sources by using data flows.
Learn more
Four new connectors available for mapping data flows (public preview)Data Factory now supports four new connectors (public preview) for mapping data flows: Quickbase connector, Smartsheet connector, TeamDesk connector, and Zendesk connector.
Learn more
Azure Cosmos DB (SQL API) for mapping data flow now supports inline modeAzure Cosmos DB (SQL API) for mapping data flow can now use inline datasets.
Learn more
Data movementGet metadata-driven data ingestion pipelines on the Data Factory Copy Data tool within 10 minutes (GA)You can build large-scale data copy pipelines with a metadata-driven approach on the Copy Data tool within 10 minutes.
Learn more
Data Factory Google AdWords connector API upgrade availableThe Data Factory Google AdWords connector now supports the new AdWords API version. No action is required for the new connector user because it's enabled by default.
Learn more
Continuous integration and continuous delivery (CI/CD)Cross tenant Azure DevOps supportConfigure a repository using Azure DevOps Git not in the same tenant as the Azure Data Factory。
Learn more
Region expansionData Factory is now available in West US3 and Jio India WestData Factory is now available in two new regions: West US3 and Jio India West. You can colocate your ETL workflow in these new regions if you're using these regions to store and manage your modern data warehouse. You can also use these regions for business continuity and disaster recovery purposes if you need to fail over from another region within the geo.
Learn more
SecurityConnect to an Azure DevOps account in another Entra ID tenantYou can connect your Data Factory instance to an Azure DevOps account in a different Azure AD tenant for source control purposes.
Learn more

January 2022

Service categoryService improvementsDetails
Data flowQuick reuse is now automatic in all Azure IRs that use Time to Live (TTL)You no longer need to manually specify "quick reuse." Data Factory mapping data flows can now start up subsequent data flow activities in under five seconds after you set a TTL.
Learn more
Retrieve your custom Assert descriptionIn the Assert transformation, you can define your own dynamic description message. You can use the new function assertErrorMessage() to retrieve the row-by-row message and store it in your destination data.
Learn more
Automatic schema detection in Parse transformationA new feature added to the Parse transformation makes it easy to automatically detect the schema of an embedded complex field inside a string column. Select the Detect schema button to set your target schema automatically.
Learn more
Support Dynamics 365 connector as both sink and sourceYou can now connect directly to Dynamics 365 to transform your Dynamics data at scale by using the new mapping data flow connector for Dynamics 365.
Learn more
Always Encrypted SQL connections now available in data flowsAlways Encrypted can now source transformations in SQL Server, SQL Database, SQL Managed Instance, and Azure Synapse when you use data flows.
Learn more
Data movementData Factory Azure Databricks Delta Lake connector supports new authentication typesData Factory Databricks Delta Lake connector now supports two more authentication types: system-assigned managed identity authentication and user-assigned managed identity authentication.
Learn more
Data Factory Copy activity supports upsert in several more connectorsData Factory Copy activity now supports upsert while it sinks data to SQL Server, SQL Database, SQL Managed Instance, and Azure Synapse.
Learn more

December 2021

Service categoryService improvementsDetails
Data flowDynamics connector as native source and sink for mapping data flowsThe Dynamics connector is now supported as source and sink for mapping data flows.
Learn more
Native change data capture (CDC) is now natively supportedCDC is now natively supported in Data Factory for Azure Cosmos DB, Blob Storage, Data Lake Storage Gen1, Data Lake Storage Gen2, and CDM.
Learn more
Flowlets public previewThe flowlets public preview allows data flow developers to build reusable components to easily build composable data transformation logic.
Learn more
Map Data public previewThe Map Data preview enables business users to define column mapping and transformations to load Azure Synapse lake databases.
Learn more
Multiple output destinations from Power QueryYou can now map multiple output destinations from Power Query in Data Factory for flexible ETL patterns for citizen data integrators.
Learn more
External Call transformation supportExtend the functionality of mapping data flows by using the External Call transformation. You can now add your own custom code as a REST endpoint or call a curated third-party service row by row.
Learn more
Enable quick reuse by Azure Synapse mapping data flows with TTL supportAzure Synapse mapping data flows now support quick reuse by setting a TTL in the Azure IR. Using a setting enables your subsequent data flow activities to execute in under five seconds.
Learn more
Assert transformationEasily add data quality, data domain validation, and metadata checks to your Data Factory pipelines by using the Assert transformation in mapping data flows.
Learn more
IntelliSense support in expression builder for more productive pipeline authoring experiencesIntelliSense support in expression builder and dynamic content authoring makes Data Factory and Azure Synapse pipeline developers more productive while they write complex expressions in their data pipelines.
Learn more

November 2021

Service categoryService improvementsDetails
Continuous integration and continuous delivery (CI/CD) GitHub integration improvements Improvements in Data Factory and GitHub integration remove limits on 1,000 Data Factory resources per resource type, such as datasets and pipelines. For large data factories, this change helps mitigate the impact of the GitHub API rate limit.
Learn more
Data flowSet a custom error code and error message with the Fail activityFail activity enables ETL developers to set the error message and custom error code for a Data Factory pipeline.
Learn more
External call transformationMapping data flows External Call transformation enables ETL developers to use transformations and data enrichments provided by REST endpoints or third-party API services.
Learn more
Synapse quick reuseWhen you execute data flow in Synapse Analytics, use the TTL feature. The TTL feature uses the quick reuse feature so that sequential data flows will execute within a few seconds. You can set the TTL when you configure an Azure IR.
Learn more
Data movementCopy activity supports reading data from FTP or SFTP without chunkingAutomatically determine the file length or the relevant offset to be read when you copy data from an FTP or SFTP server. With this capability, Data Factory automatically connects to the FTP or SFTP server to determine the file length. After the length is determined, Data Factory divides the file into multiple chunks and reads them in parallel.
Learn more
UTF-8 without BOM support in Copy activityCopy activity supports writing data with encoding the type UTF-8 without BOM for JSON and delimited text datasets.
Multicharacter column delimiter supportCopy activity supports using multicharacter column delimiters for delimited text datasets.
Integration runtime Run any process anywhere in three steps with SQL Server Integration Services (SSIS) in Data Factory Learn how to use the best of Data Factory and SSIS capabilities in a pipeline. A sample SSIS package with parameterized properties helps you get a jump-start. With Data Factory Studio, the SSIS package can be easily dragged and dropped into a pipeline and used as part of an Execute SSIS Package activity.

This capability enables you to run the Data Factory pipeline with an SSIS package on self-hosted IRs or SSIS IRs. By providing run-time parameter values, you can use the powerful capabilities of Data Factory and SSIS capabilities together. This article illustrates three steps to run any process, which can be any executable, such as an application, program, utility, or batch file, anywhere.
Learn more

October 2021

Service categoryService improvementsDetails
Data flowAzure Data Explorer and Amazon Web Services (AWS) S3 connectorsThe Microsoft Data Integration team has released two new connectors for mapping data flows. If you're using Azure Synapse, you can now connect directly to your AWS S3 buckets for data transformations. In both Data Factory and Azure Synapse, you can now natively connect to your Azure Data Explorer clusters in mapping data flows.
Learn more
Power Query activity leaves preview for GAThe Data Factory Power Query pipeline activity is now generally available. This new feature provides scaled-out data prep and data wrangling for citizen integrators inside the Data Factory browser UI for an integrated experience for data engineers. The Power Query data wrangling feature in Data Factory provides a powerful, easy-to-use pipeline capability to solve your most complex data integration and ETL patterns in a single service.
Learn more
New Stringify data transformation in mapping data flowsMapping data flows adds a new data transformation called Stringify to make it easy to convert complex data types like structs and arrays into string form. These data types then can be sent to structured output destinations.
Learn more
Integration runtime Express virtual network injection for SSIS IR (public preview) The SSIS IR now supports express virtual network injection.
Learn more:
Overview of virtual network injection for SSIS IR
Standard vs. express virtual network injection for SSIS IR
Express virtual network injection for SSIS IR
SecurityAzure Key Vault integration improvementKey Vault integration now has dropdowns so that users can select the secret values in the linked service. This capability increases productivity because users aren't required to type in the secrets, which could result in human error.
Support for user-assigned managed identity in Data FactoryCredential safety is crucial for any enterprise. The Data Factory team is committed to making the data engineering process secure yet simple for data engineers. User-assigned managed identity (preview) is now supported in all connectors and linked services that support Azure AD-based authentication.
Learn more

September 2021

Service categoryService improvementsDetails
Continuous integration and continuous deliveryExpanded CI/CD capabilitiesYou can now create a new Git branch based on any other branch in Data Factory.
Learn more
Data movementAmazon Relational Database Service (RDS) for Oracle sourcesThe Amazon RDS for Oracle sources connector is now available in both Data Factory and Azure Synapse.
Learn more
Amazon RDS for SQL Server sourcesThe Amazon RDS for the SQL Server sources connector is now available in both Data Factory and Azure Synapse.
Learn more
Support parallel copy from Azure Database for PostgreSQLThe Azure Database for PostgreSQL connector now supports parallel copy operations.
Learn more
Data flowUse Data Lake Storage Gen2 to execute pre- and post-processing commandsHadoop Distributed File System pre- and post-processing commands can now be executed by using Data Lake Storage Gen2 sinks in data flows.
Learn more
Edit data flow properties for existing instances of the Azure IR The Azure IR has been updated to allow editing of data flow properties for existing IRs. You can now modify data flow compute properties without needing to create a new Azure IR.
Learn more
TTL setting for Azure Synapse to improve pipeline activities execution startup timeAzure Synapse has added TTL to the Azure IR to enable your data flow pipeline activities to begin execution in seconds, which greatly minimizes the runtime of your data flow pipelines.
Learn more
Integration runtimeData Factory managed virtual network GAYou can now provision the Azure IR as part of a managed virtual network and use private endpoints to securely connect to supported data stores. Data traffic goes through Azure Private Links, which provides secured connectivity to the data source. It also prevents data exfiltration to the public internet.
Learn more
OrchestrationOperationalize and provide SLA for data pipelinesThe new Elapsed Time Pipeline Run metric, combined with Data Factory alerts, empowers data pipeline developers to better deliver SLAs to their customers. Now you can tell us how long a pipeline should run, and we'll notify you proactively when the pipeline runs longer than expected.
Learn more
Fail activity (public preview)The new Fail activity allows you to throw an error in a pipeline intentionally for any reason. For example, you might use the Fail activity if a Lookup activity returns no matching data or a custom activity finishes with an internal error.
Learn more

August 2021

Service categoryService improvementsDetails
Continuous integration and continuous deliveryCI/CD improvements with GitHub support in Azure Government and Microsoft Azure operated by 21VianetWe've added support for GitHub in Azure for US Government and Azure operated by 21Vianet.
Learn more
Data movementThe Azure Cosmos DB API for MongoDB connector supports versions 3.6 and 4.0 in Data FactoryThe Data Factory Azure Cosmos DB API for MongoDB connector now supports server versions 3.6 and 4.0.
Learn more
Enhance using COPY statement to load data into Azure SynapseThe Data Factory Azure Synapse connector now supports staged copy and copy source with *.* as wildcardFilename for the COPY statement.
Learn more
Data flowREST endpoints are available as source and sink in data flowData flows in Data Factory and Azure Synapse now support REST endpoints as both a source and sink with full support for both JSON and XML payloads.
Learn more
Integration runtimeDiagnostic tool is available for self-hosted IRA diagnostic tool for self-hosted IR is designed to provide a better user experience and help users to find potential issues. The tool runs a series of test scenarios on the self-hosted IR machine. Every scenario has typical health check cases for common issues.
Learn more
OrchestrationCustom event trigger with advanced filtering option GAYou can now create a trigger that responds to a custom topic posted to Azure Event Grid. You can also use advanced filtering to get fine-grain control over what events to respond to.
Learn more

July 2021

Service categoryService improvementsDetails
Data movementGet metadata-driven data ingestion pipelines on the Data Factory Copy Data tool within 10 minutes (public preview)Now you can build large-scale data copy pipelines with a metadata-driven approach on the Copy Data tool (public preview) within 10 minutes.
Learn more
Data flowNew map functions added in data flow transformation functionsA new set of data flow transformation functions enables data engineers to easily generate, read, and update map data types and complex map structures.
Learn more
Integration runtimeFive new regions are available in Data Factory managed virtual network (public preview)Five new regions, China East2, China North2, US Government Arizona, US Government Texas, and US Government Virginia, are available in the Data Factory managed virtual network (public preview).
Developer productivityData Factory home page improvementsThe Data Factory home page has been redesigned with better contrast and reflow capabilities. A few sections are introduced on the home page to help you improve productivity in your data integration journey.
Learn more
New landing page for Data Factory StudioThe landing page for the Data Factory pane in the Azure portal.
Learn more

June 2021

Service categoryService improvementsDetails
Data movementNew user experience with Data Factory Copy Data toolThe redesigned Copy Data tool is now available with improved data ingestion experience.
Learn more
MongoDB and MongoDB Atlas are supported as both source and sinkThis improvement supports copying data between any supported data store and MongoDB or MongoDB Atlas database.
Learn more
Always Encrypted is supported for SQL Database, SQL Managed Instance, and SQL Server connectors as both source and sinkAlways Encrypted is available in Data Factory for SQL Database, SQL Managed Instance, and SQL Server connectors for the Copy activity.
Learn more
Setting custom metadata is supported in Copy activity when sinking to Data Lake Storage Gen2 or Blob StorageWhen you write to Data Lake Storage Gen2 or Blob Storage, the Copy activity supports setting custom metadata or storage of the source file's last modified information as metadata.
Learn more
Data flowSQL Server is now supported as a source and sink in data flowsSQL Server is now supported as a source and sink in data flows. Follow the link for instructions on how to configure your networking by using the Azure IR managed virtual network feature to talk to your SQL Server on-premises and cloud VM-based instances.
Learn more
Dataflow Cluster quick reuse now enabled by default for all new Azure IRsThe popular data flow quick startup reuse feature is now generally available for Data Factory. All new Azure IRs now have quick reuse enabled by default.
Learn more
Power Query (public preview) activityYou can now build complex field mappings to your Power Query sink by using Data Factory data wrangling. The sink is now configured in the pipeline in the Power Query (public preview) activity to accommodate this update.
Learn more
Updated data flows monitoring UI in Data FactoryData Factory has a new update for the monitoring UI to make it easier to view your data flow ETL job executions and quickly identify areas for performance tuning.
Learn more
SQL Server Integration ServicesRun any SQL statements or scripts anywhere in three steps with SSIS in Data FactoryThis post provides three steps to run any SQL statements or scripts anywhere with SSIS in Data Factory.
  1. Prepare your self-hosted IR or SSIS IR.
  2. Prepare an Execute SSIS Package activity in Data Factory pipeline.
  3. Run the Execute SSIS Package activity on your self-hosted IR or SSIS IR.
Learn more

More information