How do I parameterize key vaults and secrets in an ADO release pipeline for ADF

Diana Rus 25 Reputation points
2024-09-03T13:43:04.75+00:00

Hi,

I am building a release pipeline in Azure DevOps to deploy from my Dev ADF into my Prod ADF. A lot of my linked services use a key vault with a secret to connect, however this is not exported in the ARM template parameters so I can parameterize in the release pipeline and override. How can I do this as the key vault in dev is different to the key vault in prod.

Thank you

Diana

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,566 questions
{count} votes

Accepted answer
  1. Smaran Thoomu 14,870 Reputation points Microsoft Vendor
    2024-09-03T22:28:45.67+00:00

    Hi @Diana Rus
    Thanks for the question and using MS Q&A platform.
    By default all artifacts properties are not exposed and parameterized.

    Once you integrate your workspace with your Git repository and publish the changes, you will see the below two files.

    TemplateForWorkspace.json is the ARM template containing all the workspace artifacts and resources

    TemplateParametersForWorkspace.json is the ARM template containing only the artifacts parameters.

    TemplateParametersForWorkspace.json file containing a global parameter for your workspace name and a parameter for each workspace default linked service:

    for the artifacts properties that are not exposed by default, you will need to create the arm-template-parameters-definition.json under the main root folder and parameterize the artifacts properites.

    You can follow the below document for this.

    https://video2.skills-academy.com/en-us/azure/data-factory/continuous-integration-delivery-resource-manager-custom-parameters#custom-parameter-syntax

    By using the custom template parameter file, you can automate your CICD process in ADF to override any artifact property that is not parameterized by the default parameters template.

    once the values are parameterized then you can directly use them on the release pipeline on the overrideparametes section.

    Here is an example for sample parameterization template:

      
    
        "Microsoft.DataFactory/factories/linkedServices": {         
    		"*": {             
    			"properties": {                 
    				"typeProperties": {                     
    					"accountName": "=",                     
    					"username": "=",                     
    					"connectionString": "|:-connectionString:secureString",                     "secretAccessKey": "|"                 
    					}             
    				}         
    			}
    

    the above code will parameterize all of the parameters in all of the pipelines.

    • Linked services are unique. Because linked services and datasets have a wide range of types, you can provide type-specific customization. In this example, for all linked services of type AzureDataLakeStore, a specific template will be applied. For all others (via *), a different template will be applied.
    • The connectionString property will be parameterized as a securestring value. It won't have a default value. It will have a shortened parameter name that's suffixed with connectionString.
    • The property secretAccessKey happens to be an AzureKeyVaultSecret (for example, in an Amazon S3 linked service). It's automatically parameterized as an Azure Key Vault secret and fetched from the configured key vault. You can also parameterize the key vault itself.

    You can modify this based on your requirement.

    I hope this answers your question. Please let us know if you have any further questions.


0 additional answers

Sort by: Most helpful

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.