ADF-Global parameters inside Linked Service

Dario Antolini 21

Hi all!

My company is starting to make use of DevOps pipeline to share the same code in all the productive and not productive envirorments. Before this, everything was wild. For example, the linked service for the Azure KeyVault has the url-link written inside it, and now any time we deploy this solution we need to modify the url in every other envirorments.

We tried to create a global parameter for this url and a parameter-value inside the LS of AzureKeyVault, but this will affect ALL the other linked services, datasets and pipeline activity, and since we already have A LOT OF pipelines we don't want to use this solution.

Is there a way to put the KeyVault-url global parameter directly inside the relative Linked Service? Or is there a way to modify the Azure DevOps pipeline in order to modify this url inside the KeyVault linked service?

Thank you,

Vinodh247 16,751 Reputation points

2023-10-01T12:29:06.9333333+00:00

Hi Dario Antolini:

Thanks for reaching out to Microsoft Q&A.

You can parameterize linked services using 'add dynamic content' option like given below. Let me know if this is what you are looking for.

https://video2.skills-academy.com/en-us/azure/data-factory/parameterize-linked-services?tabs=data-factory

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.
Dario Antolini 21 Reputation points

2023-10-01T12:46:48.5033333+00:00

hi @Vinodh247 , thank you for your answer.

I know I can parametrize my linked service, but doing this we are forced to modify the already existing pipeline activities. Because the linked service I am interesting in is versus Azure KeyVault, and so ALL the activities requesting any type of connections (ADLS, AzureSQL db, ecc..) pass through the Azure KeyVault and then they must be modified to add the value for the KeyVault url.

Since there are tons of these activities, we don't want to proced in this direction.

Our situation is the following: we have a Linked Service for Azure KeyVault, which is called by all other Linked Services (e.g. ADLS, Azure SQL db, Azure Databricks...).

Is this possibile to insert a global parameter at the linked service level rather than at the pipeline activity level?

Thank you in advance!
Harishga 5,990 Reputation points Microsoft Vendor

2023-10-02T13:22:17.5933333+00:00
Hi@Dario Antolini ,

Welcome to Microsoft Q&A platform and thanks for posting your question here.

To set up a global parameter for the URL of an Azure Key Vault linked service in Azure Data Factory (ADF) to avoid modifying the URL in every environment when deploying the solution, you can follow these steps:

Create a global parameter in the pipeline's parameters section. To create a global parameter, go to the Global parameters tab in the Manage section. Select New to open the creation side-nav. In the side-nav, enter a name, select a data type, and specify the value of your parameter.

Use the ADF's expression language to reference the global parameter in the linked service's URL property. The expression language allows you to reference the global parameter in the linked service's URL property, which will be replaced with the parameter value at runtime.

Here's an example of how you can reference the global parameter in the linked service's URL property:
@concat('https://', pipeline().globalParameters.KeyVaultUrl, '.vault.azure.net/')
In this example, pipeline().globalParameters.KeyVaultUrl references the global parameter named KeyVaultUrl, which you can define in the pipeline's parameters section. The concat function concatenates the https:// prefix, the global parameter value, and the .vault.azure.net/ suffix to form the complete URL.

You can then use this expression in the linked service's URL property, as shown below:

"parameters": { "KeyVaultUrl": { "type": "string", "defaultValue": "mykeyvault" } }, "properties": { "type": "AzureKeyVault", "typeProperties": { "baseUrl": "@concat('https://', pipeline().globalParameters.KeyVaultUrl, '.vault.azure.net/')", "authenticationType": "ManagedServiceIdentity" } }

In this example, the baseUrl property references the expression that concatenates the global parameter value with the URL prefix and suffix.

Reference: https://video2.skills-academy.com/en-us/azure/data-factory/parameterize-linked-services

I hope this information helps you. Let me know if you have any further questions or concerns.
Harishga 5,990 Reputation points Microsoft Vendor

2023-10-03T13:45:30.2+00:00

Hi@Dario Antolini ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Harishga 5,990 Reputation points Microsoft Vendor

2023-10-04T14:23:05.1933333+00:00

Hi@Dario Antolini ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Dario Antolini 21 Reputation points

2023-10-09T20:48:43.1033333+00:00

Thank you very much for you help.

The solution with the parameter valuable at runtime is good, but it does not fit our needs. Since we want something that it can be tested for other Linked Services and Datasets, we preferred to use a different approach.

Our CI/CD pipeline starts only in Dev envirorment, and every release can deploy the same code in the other envirorment. The problem is in the fundational linked service for Azure KeyVault because it has inside a direct URL pointing to the Azure KeyVault for Dev env. When we deploy a release in other envirorments, this URL remains pointing to the one of Dev. So we have decided to add another step in the release section of Azure DevOps, where it calls the API-PUT for updating this linked service, so that we can change the URL depending on the envirorment the release is deploying.

In this way, ADF architecture remains the same as we have right now, and after the release we get the right URL for the Azure KeyVault of the relative envirorment.

Hope this can help someone else.

Dario
Harishga 5,990 Reputation points Microsoft Vendor

2023-10-10T07:13:32.9466667+00:00

@Dario Antolini I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others "
Mehul Gardi 0 Reputation points

2024-07-17T01:21:26.5766667+00:00

We have a similar need for parameterizing the host for SFTP linked Service. We did the steps as above

We created a global Parameter for the host named SFTPLocation.

When we are trying to use the expression @{pipeline().globalParameters.SFTPLocation} or @{concat('',pipeline().globalParameters.SFTPLocation)} , we are getting the error as The expression {} cannot be evaluated because property 'globalParameters' doesn't exist . Can it be confirmed that the Host property of SFTP linked service supports dynamic expression. We want to do it for Host as well as the UserName since they do not have KV integration as Password
Vinodh247 16,751 Reputation points

2024-07-17T05:05:15.74+00:00

Mehul Gardi: Pls raise a separate thread for your question
Mathias Denzin 0 Reputation points

2024-08-30T01:40:05.0066667+00:00
@harishgada When I try what you explained with a Salesforce connection, and I just tried one parameter as a test, it does not work.

I created the global parameter "sfdcAPIversion" as a string and saved the value "54.0". Then I activated "Specify dynamic contents in JSON format" and set up the following JSON configuration:

{ "properties": { "type": "SalesforceV2", "typeProperties": { "apiVersion": "@pipeline().globalParameters.sfdAPIversion" } } }

When I now try to test the connection, I get the following error:

The parameters and expression cannot be resolved for schema operations. Error Message: {

"message": "ErrorCode=InvalidTemplate, ErrorMessage=The expression 'pipeline().globalParameters.sfdAPIversion' cannot be evaluated because property 'globalParameters' doesn't exist, available properties are ''.."

}

Activity ID: e0a85bab-34d0-42bc-9390-eee7a136b5e9

1 answer

PRADEEPCHEEKATLA-MSFT 88,461 Reputation points Microsoft Employee

2023-10-10T12:21:45.8566667+00:00

@Dario Antolini - I'm glad that you were able to resolve your issue and thank you for posting your solution so that others experiencing the same thing can easily reference this! Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your solution in case you'd like to accept the answer .

Ask: ADF-Global parameters inside Linked Service

Is there a way to put the KeyVault-url global parameter directly inside the relative Linked Service? Or is there a way to modify the Azure DevOps pipeline in order to modify this url inside the KeyVault linked service?

Solution: The solution with the parameter valuable at runtime is good, but it does not fit our needs. Since we want something that it can be tested for other Linked Services and Datasets, we preferred to use a different approach.

Our CI/CD pipeline starts only in Dev envirorment, and every release can deploy the same code in the other envirorment. The problem is in the fundational linked service for Azure KeyVault because it has inside a direct URL pointing to the Azure KeyVault for Dev env. When we deploy a release in other envirorments, this URL remains pointing to the one of Dev. So we have decided to add another step in the release section of Azure DevOps, where it calls the API-PUT for updating this linked service, so that we can change the URL depending on the envirorment the release is deploying.

In this way, ADF architecture remains the same as we have right now, and after the release we get the right URL for the Azure KeyVault of the relative envirorment.

If I missed anything please let me know and I'd be happy to add it to my answer, or feel free to comment below with any additional information.

I hope this helps!

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

ADF-Global parameters inside Linked Service

1 answer

Your answer