Use Azure Key Vault secrets in Azure Pipelines

Azure DevOps Services | Azure DevOps Server 2022 - Azure DevOps Server 2019

Azure Key Vault allows developers to securely store and manage sensitive information like API keys, credentials, or certificates. Azure Key Vault service supports two types of containers: vaults and managed HSM (Hardware Security Module) pools. Vaults can store both software and HSM-backed keys, secrets, and certificates, while managed HSM pools exclusively support HSM-backed keys.

In this tutorial, you will learn how to:

  • Create an Azure Key Vault using Azure CLI
  • Add a secret and configure access to Azure key vault
  • Use secrets in your pipeline

Prerequisites

Create a repo

If you already have your own repository, proceed to the next step. Otherwise, follow the instructions below to initialize your repository. We will use this Azure Repo to set up our pipeline.

  1. Sign in to your Azure DevOps organization, and then navigate to your project.

  2. Select Repos, and then select Initialize to initialize the main branch with a README.

    A screenshot showing how to initialize a repository with a README file.

Create an Azure Key Vault

  1. Sign in to the Azure Portal, and then select the Cloud Shell button in the upper-right corner.

  2. If you have more than one Azure subscription associated with your account, use the command below to specify a default subscription. You can use az account list to generate a list of your subscriptions.

    az account set --subscription <YOUR_SUBSCRIPTION_NAME_OR_ID>
    
  3. Set your default Azure region. You can use az account list-locations to generate a list of available regions.

    az config set defaults.location=<YOUR_REGION>
    
  4. Create a new resource group.

    az group create --name <YOUR_RESOURCE_GROUP_NAME>
    
  5. Create a new Azure Key Vault.

    az keyvault create \
      --name <YOUR_KEY_VAULT_NAME> \
      --resource-group <YOUR_RESOURCE_GROUP_NAME>
    
  6. Create a new secret in your Azure key vault.

    az keyvault secret set \
      --name <YOUR_SECRET_NAME> \
      --value <YOUR_ACTUAL_SECRET> \
      --vault-name <YOUR_KEY_VAULT_NAME>
    

Set up key vault access policies

To access our Azure Key Vault, we need to set up a service principal to grant access to Azure Pipelines. Follow this guide to create a service principal with Azure CLI, and then continue with the next steps in this section.

  1. Navigate to Azure portal, and then use the search bar to find the key vault you created earlier.

    A screenshot showing how to search for your Azure Key Vault.

  2. Select Access policies, and then select Create to create a new policy.

  3. Under Secret permissions, select Get and List.

  4. Select Next, and then select the service principal you created earlier. A service principal is an object that represents an application or service that's requesting access to Azure resources.

  5. Select Next, and then Next once more.

  6. Review your policies, and then select Create when you're done.

Add role assignment

In the next step, we'll create an ARM service connection using service principal. Before we can verify the connection, we need to grant the service principal Read access at the subscription level:

  1. Navigate to Azure portal

  2. Select Subscriptions from the left navigation panel, and then find and select your subscription.

  3. Select Access control, and then select Add > Add role assignment.

    A screenshot showing how to add a new role assignment at the subscription level.

  4. Select Reader under the Role tab, and then select Next.

  5. Select User, group, or service principal, and then select Select members.

    A screenshot showing how to add members to role assignment in Azure.

  6. Use the search bar to find your service principal, and then select the "+" sign to select it, then click on the Select button.

  7. Select Review + assign, review your settings, and then select Review + assign once more to confirm your choices and add the role assignment.

Create a service connection

  1. Sign in to your Azure DevOps organization, and then navigate to your project.

  2. Select Project settings > Service connections, and then select New service connection to create a new service connection.

  3. Select Azure Resource Manager, and then select Next.

  4. Select Service principal (manual), and then select Next.

  5. Select Azure Cloud for Environment and Subscription for the Scope Level, then enter your Subscription Id and your Subscription Name.

  6. Fill out the following fields with the information you obtained when creating the service principal, and then select Verify when you're done:

    • Service Principal Id: Your service principal appId.
    • Service Principal key: Your service principal password.
    • Tenant ID: Your service principal tenant.
  7. Once the verification has succeeded, provide a name and description (optional) for your service connection, and then check the Grant access permission to all pipelines checkbox.

  8. Select Verify and save when you're done.

    A screenshot showing how to create a new Azure Resource Manager service connection using service principal.

  1. Sign in to your Azure DevOps collection, and then navigate to your project.

  2. Select Project settings > Service connections > New service connection and then select Azure Resource Manager to create a new ARM service connection.

  3. Give your service connection a name, and then select Azure Cloud for Environment and Subscription for the Scope Level.

  4. Enter your Subscription Id and your Subscription Name.

  5. Fill out the following fields with the information you obtained when creating the service principal, and then select Verify connection when you're done:

    • Service Principal client Id: Your service principal appId.
    • Service Principal key: Your service principal password.
    • Tenant ID: Your service principal tenant.
  6. Check the Allow all pipelines to use this connection checkbox, and then select Ok when you're done.

    A screenshot showing how to create a new ARM service connection using service principal in Azure DevOps Server 2019.

Create a new pipeline

  1. Sign in to your Azure DevOps organization, and then navigate to your project.

  2. Select Pipelines, and then select New Pipeline.

  3. Select Use the classic editor to create a classic pipeline.

  4. Select Azure Repos Git, select your repository and default branch, and then select Continue.

  5. Select the .Net Desktop pipeline template.

  6. For this example, we will only need the last two tasks. Press CTRL and then select the first five tasks, right-click and choose Remove selected tasks(s) to delete them.

    A screenshot showing how to delete multiple pipeline tasks.

  7. Select + to add a new task. Search for the Command line task, select it, and then select Add to add it to your pipeline. Once added, configure it as follows:

    • Display name: Create file
    • Script: echo $(YOUR_SECRET_NAME) > secret.txt

    A screenshot showing how to configure the command line task.

  8. Select + to add a new task. Search for the Azure Key Vault task, select it, and then select Add* to add it to your pipeline. Once added, configure it as follows:

    • Display name: Azure Key Vault
    • Azure subscription: select your service principal service connection you created earlier
    • Key vault: select your key vault
    • Secret filter: A comma separated list of secret names or leave * to download all secrets from the selected key vault

    A screenshot showing how to set up the Azure Key Vault task in classic pipelines.

  9. Select the Copy files task and fill out the required fields as follows:

    • Display name: Copy File
    • Contents: secret.txt
    • Target Folder: $(build.artifactstagingdirectory)

    A screenshot showing how to set up the copy files task in classic pipelines.

  10. Select the Publish Artifacts task and fill out the required fields as follows:

    • Display name: Publish Artifact
    • Path to publish: $(build.artifactstagingdirectory)
    • Artifact name: drop
    • Artifact publish location: Azure Pipelines

    A screenshot showing how to set up the publish artifacts task in classic pipelines.

  11. Select Save and queue, and then select Run to run your pipeline.

  12. Once the pipeline run is complete, return to the pipeline summary and select the published artifact.

  13. Select drop > secret.txt to download the published artifact.

    A screenshot showing how to download the published artifact.

  14. Open the text file you just downloaded, the text file should contain the secret from your Azure key vault.

  1. Sign in to your Azure DevOps collection, and then navigate to your project.

  2. Select Pipelines, and then select Builds.

  3. Select New > New build pipeline.

  4. Select Use the classic editor to create a new classic build pipeline.

  5. Select Azure Repos Git, select your repository and your default branch, and then select Continue.

  6. Select the .Net Desktop pipeline template, and then select Apply.

  7. For this example, we will only need the last two tasks. Press CTRL and then select the first five tasks, right-click and choose Remove selected tasks(s) to delete them.

    A screenshot showing how to delete multiple pipeline tasks in classic pipelines in Azure DevOps Server 2019.

  8. Select + to add a new task. Search for the Command line task, select it, and then select Add to add it to your pipeline. Once added, configure it as follows:

    • Display name: Create file
    • Script: echo $(YOUR_SECRET_NAME) > secret.txt

    A screenshot showing how to configure the command line task in classic pipelines in Azure DevOps Server 2019.

  9. Select + to add a new task. Search for the Azure Key Vault task, select it, and then select Add* to add it to your pipeline. Once added, configure it as follows:

    • Display name: Azure Key Vault
    • Azure subscription: select your service principal service connection you created earlier
    • Key vault: select your key vault
    • Secret filter: A comma separated list of secret names or leave * to download all secrets from the selected key vault

    A screenshot showing how to set up the Azure Key Vault task in a classic pipeline in Azure DevOps Server 2019.

  10. Select the Copy files task and fill out the required fields as follows:

    • Display name: Copy File
    • Contents: secret.txt
    • Target Folder: $(build.artifactstagingdirectory)

    A screenshot showing how to set up the copy files task in classic pipelines in Azure DevOps Server 2019.

  11. Select the Publish Artifacts task and fill out the required fields as follows:

    • Display name: Publish Artifact
    • Path to publish: $(build.artifactstagingdirectory)
    • Artifact name: drop
    • Artifact publish location: Azure Pipelines

    A screenshot showing how to set up the publish artifacts task in classic pipelines in Azure DevOps Server 2019.

  12. Select Save & queue, and then select Save & queue to run your build pipeline.

  13. Once the pipeline run is complete, select Artifacts and then select drop.

  14. In the newly opened window, select drop > secret.txt, select the ellipsis icon (...), and then select download to save the text file.

  15. Open the text file you just downloaded, it should contain the secret from your Azure key vault.

Warning

This tutorial is for educational purposes only. For security best practices and how to safely work with secrets, see Manage secrets in your server apps with Azure Key Vault.

Clean up resources

Follow the steps below to delete the resources you created:

  1. If you've created a new organization to host your project, see how to delete your organization, otherwise delete your project.

  2. All Azure resources created during this tutorial are hosted under a single resource group. Run the following command to delete your resource group and all of its resources.

    az group delete --name <YOUR_RESOURCE_GROUP_NAME>
    

FAQ

Q: I'm getting the following error: "the user or group does not have secrets list permission" what should I do?

A: If you encounter an error indicating that the user or group does not have secrets list permission on key vault, run the following commands to authorize your application to access the key or secret in the Azure Key Vault:

$ErrorActionPreference="Stop";
$Credential = Get-Credential;
Connect-AzAccount -SubscriptionId <YOUR_SUBSCRIPTION_ID> -Credential $Credential;
$spn=(Get-AzureRmADServicePrincipal -SPN <YOUR_SERVICE_PRINCIPAL_ID>);
$spnObjectId=$spn.Id;
Set-AzureRmKeyVaultAccessPolicy -VaultName key-vault-tutorial -ObjectId $spnObjectId -PermissionsToSecrets get,list;