Create a shared self-hosted integration runtime in Azure Data Factory

APPLIES TO: Azure Data Factory Azure Synapse Analytics

This guide shows you how to create a shared self-hosted integration runtime in Azure Data Factory. Then you can use the shared self-hosted integration runtime in another data factory.

Create a shared self-hosted integration runtime in Azure Data Factory

You can reuse an existing self-hosted integration runtime infrastructure that you already set up in a data factory. This reuse lets you create a linked self-hosted integration runtime in a different data factory by referencing an existing shared self-hosted IR.

To see an introduction and demonstration of this feature, watch the following 12-minute video:

Terminology

• Shared IR: An original self-hosted IR that runs on a physical infrastructure.
• Linked IR: An IR that references another shared IR. The linked IR is a logical IR and uses the infrastructure of another shared self-hosted IR.

Create a shared self-hosted IR using Azure Data Factory UI

To create a shared self-hosted IR using Azure Data Factory UI, you can take following steps:

1. In the self-hosted IR to be shared, select Grant permission to another Data factory and in the "Integration runtime setup" page, select the Data factory in which you want to create the linked IR.

   

2. Note and copy the above "Resource ID" of the self-hosted IR to be shared.

3. In the data factory to which the permissions were granted, create a new self-hosted IR (linked) and enter the resource ID.

   

   

   

Create a shared self-hosted IR using Azure PowerShell

To create a shared self-hosted IR using Azure PowerShell, you can take following steps:

1. Create a data factory.
2. Create a self-hosted integration runtime.
3. Share the self-hosted integration runtime with other data factories.
4. Create a linked integration runtime.
5. Revoke the sharing.

Prerequisites

• Azure subscription. If you don't have an Azure subscription, create a free account before you begin.

• Azure PowerShell. Follow the instructions in Install Azure PowerShell on Windows with PowerShellGet. You use PowerShell to run a script to create a self-hosted integration runtime that can be shared with other data factories.

Create a data factory

1. Launch the Windows PowerShell Integrated Scripting Environment (ISE).

2. Create variables. Copy and paste the following script. Replace the variables, such as SubscriptionName and ResourceGroupName, with actual values:

   # If input contains a PSH special character, e.g. "$", precede it with the escape character "`" like "`$". 
   $SubscriptionName = "[Azure subscription name]" 
   $ResourceGroupName = "[Azure resource group name]" 
   $DataFactoryLocation = "EastUS" 
   
   # Shared Self-hosted integration runtime information. This is a Data Factory compute resource for running any activities 
   # Data factory name. Must be globally unique 
   $SharedDataFactoryName = "[Shared Data factory name]" 
   $SharedIntegrationRuntimeName = "[Shared Integration Runtime Name]" 
   $SharedIntegrationRuntimeDescription = "[Description for Shared Integration Runtime]"
   
   # Linked integration runtime information. This is a Data Factory compute resource for running any activities
   # Data factory name. Must be globally unique
   $LinkedDataFactoryName = "[Linked Data factory name]"
   $LinkedIntegrationRuntimeName = "[Linked Integration Runtime Name]"
   $LinkedIntegrationRuntimeDescription = "[Description for Linked Integration Runtime]"
   

3. Sign in and select a subscription. Add the following code to the script to sign in and select your Azure subscription:

   Connect-AzAccount
   Select-AzSubscription -SubscriptionName $SubscriptionName
   

4. Create a resource group and a data factory.

   Note

   This step is optional. If you already have a data factory, skip this step.

   Create an Azure resource group by using the New-AzResourceGroup command. A resource group is a logical container into which Azure resources are deployed and managed as a group. The following example creates a resource group named myResourceGroup in the WestEurope location:

   New-AzResourceGroup -Location $DataFactoryLocation -Name $ResourceGroupName
   

   Run the following command to create a data factory:

   Set-AzDataFactoryV2 -ResourceGroupName $ResourceGroupName `
                            -Location $DataFactoryLocation `
                            -Name $SharedDataFactoryName
   

Create a self-hosted integration runtime

Run the following command to create a self-hosted integration runtime:

$SharedIR = Set-AzDataFactoryV2IntegrationRuntime `
    -ResourceGroupName $ResourceGroupName `
    -DataFactoryName $SharedDataFactoryName `
    -Name $SharedIntegrationRuntimeName `
    -Type SelfHosted `
    -Description $SharedIntegrationRuntimeDescription

Get the integration runtime authentication key and register a node

Run the following command to get the authentication key for the self-hosted integration runtime:

Get-AzDataFactoryV2IntegrationRuntimeKey `
    -ResourceGroupName $ResourceGroupName `
    -DataFactoryName $SharedDataFactoryName `
    -Name $SharedIntegrationRuntimeName

The response contains the authentication key for this self-hosted integration runtime. You use this key when you register the integration runtime node.

Install and register the self-hosted integration runtime

1. Download the self-hosted integration runtime installer from Azure Data Factory Integration Runtime.

2. Run the installer to install the self-hosted integration on a local computer.

3. Register the new self-hosted integration with the authentication key that you retrieved in a previous step.

Share the self-hosted integration runtime with another data factory

Create another data factory

$factory = Set-AzDataFactoryV2 -ResourceGroupName $ResourceGroupName `
    -Location $DataFactoryLocation `
    -Name $LinkedDataFactoryName

Grant permission

Grant permission to the data factory that needs to access the self-hosted integration runtime you created and registered.

New-AzRoleAssignment `
    -ObjectId $factory.Identity.PrincipalId ` #MSI of the Data Factory with which it needs to be shared
    -RoleDefinitionName 'Contributor' `
    -Scope $SharedIR.Id

Create a linked self-hosted integration runtime

Run the following command to create a linked self-hosted integration runtime:

Set-AzDataFactoryV2IntegrationRuntime `
    -ResourceGroupName $ResourceGroupName `
    -DataFactoryName $LinkedDataFactoryName `
    -Name $LinkedIntegrationRuntimeName `
    -Type SelfHosted `
    -SharedIntegrationRuntimeResourceId $SharedIR.Id `
    -Description $LinkedIntegrationRuntimeDescription

Now you can use this linked integration runtime in any linked service. The linked integration runtime uses the shared integration runtime to run activities.

Revoke integration runtime sharing from a data factory

To revoke the access of a data factory from the shared integration runtime, run the following command:

Remove-AzRoleAssignment `
    -ObjectId $factory.Identity.PrincipalId `
    -RoleDefinitionName 'Contributor' `
    -Scope $SharedIR.Id

To remove the existing linked integration runtime, run the following command against the shared integration runtime:

Remove-AzDataFactoryV2IntegrationRuntime `
    -ResourceGroupName $ResourceGroupName `
    -DataFactoryName $SharedDataFactoryName `
    -Name $SharedIntegrationRuntimeName `
    -LinkedDataFactoryName $LinkedDataFactoryName

Monitoring

Shared IR

Linked IR

Known limitations of self-hosted IR sharing

• The data factory in which a linked IR is created must have an Managed Identity. By default, the data factories created in the Azure portal or PowerShell cmdlets have an implicitly created Managed Identity. But when a data factory is created through an Azure Resource Manager template or SDK, you must set the Identity property explicitly. This setting ensures that Resource Manager creates a data factory that contains a Managed Identity.

• The Data Factory .NET SDK that supports this feature must be version 1.1.0 or later.

• To grant permission, you need the Owner role or the inherited Owner role in the data factory where the shared IR exists.

• The sharing feature works only for data factories within the same Microsoft Entra tenant.

• For Microsoft Entra ID guest users, the search functionality in the UI, which lists all data factories by using a search keyword, doesn't work. But as long as the guest user is the owner of the data factory, you can share the IR without the search functionality. For the Managed Identity of the data factory that needs to share the IR, enter that Managed Identity in the Assign Permission box and select Add in the Data Factory UI.

  Note

  This feature is available only in Data Factory V2.

Related content

• Review integration runtime concepts in Azure Data Factory.

• Learn how to create a self-hosted integration runtime in the Azure portal.