Use the Secret Store extension to fetch secrets for offline access in Azure Arc-enabled Kubernetes clusters

The Azure Key Vault Secret Store extension for Kubernetes ("Secret Store") automatically synchronizes secrets from an Azure Key Vault to an Azure Arc-enabled Kubernetes cluster for offline access. This means you can use Azure Key Vault to store, maintain, and rotate your secrets, even when running your Kubernetes cluster in a semi-disconnected state. Synchronized secrets are stored in the cluster secret store, making them available as Kubernetes secrets to be used in all the usual ways: mounted as data volumes, or exposed as environment variables to a container in a pod.

Synchronized secrets are critical business assets, so the Secret Store secures them through isolated namespaces and nodes, role-based access control (RBAC) policies, and limited permissions for the secrets synchronizer. For extra protection, encrypt the Kubernetes secret store on your cluster.

This article shows you how to install and configure the Secret Store as an Azure Arc-enabled Kubernetes extension.

Prerequisites

• A cluster connected to Azure Arc, running Kubernetes version 1.27 or higher, and in one of the supported regions (East US, East US2, West US, West US2, West US3, West Europe, North Europe). The region is defined by the resource group region used for creating the Arc cluster.
• The examples throughout this guide use a K3s cluster.
• Ensure you meet the general prerequisites for cluster extensions, including the latest version of the k8s-extension Azure CLI extension.
• cert-manager is required to support TLS for intracluster log communication. The examples later in this guide direct you though installation. For more information about cert-manager, see cert-manager.io

Before you begin, set environment variables to be used for configuring Azure and cluster resources. If you already have a managed identity, Azure Key Vault, or other resource listed here, update the names in the environment variables to reflect those resources.

az login
export RESOURCE_GROUP="oidc-issuer"
export LOCATION="westus2"
export AZURE_STORAGE_ACCOUNT="oidcissuer$(openssl rand -hex 4)"
export AZURE_STORAGE_CONTAINER="oidc-test"
export SUBSCRIPTION="$(az account show --query id --output tsv)"
export AZURE_TENANT_ID="$(az account show -s $SUBSCRIPTION --query tenantId --output tsv)"
export CURRENT_USER="$(az ad signed-in-user show --query userPrincipalName --output tsv)"
export KEYVAULT_NAME="azwi-kv-$(openssl rand -hex 4)"
export KEYVAULT_SECRET_NAME="my-secret"
export USER_ASSIGNED_IDENTITY_NAME="myIdentity"
az account set --subscription "${SUBSCRIPTION}"
export SERVICE_ACCOUNT_ISSUER="https://$AZURE_STORAGE_ACCOUNT.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}"
export FEDERATED_IDENTITY_CREDENTIAL_NAME="myFedIdentity"
export KUBERNETES_NAMESPACE="default"
export SERVICE_ACCOUNT_NAME="workload-identity-sa"

Configure an identity to access secrets

To access and synchronize a given Azure Key Vault secret, the Secret Store requires access to an Azure managed identity with appropriate Azure permissions to access that secret. The managed identity must be linked to a Kubernetes service account through federation. The Kubernetes service account is what you use in a Kubernetes pod or other workload to access secrets from the Kubernetes secret store. The Secret Store extension uses the associated federated Azure managed identity to pull secrets from Azure Key Vault to your Kubernetes secret store. The following sections describe how to set this up.

Host OIDC public information about your cluster's Service Account issuer

Use of federated identity currently requires you to set up cloud storage to host OIDC format information about the public keys of your cluster's Service Account issuer. In this section, you set up a secured, public Open ID Connect (OIDC) issuer URL using Azure blob storage, then upload a minimal discovery document to the storage account. For background, see OIDC configuration for self-managed clusters.

1. Create an Azure storage account.

   az group create --name "${RESOURCE_GROUP}" --location "${LOCATION}"
   az storage account create --resource-group "${RESOURCE_GROUP}" --name "${AZURE_STORAGE_ACCOUNT}" --allow-blob-public-access true
   az storage container create --name "${AZURE_STORAGE_CONTAINER}" --public-access blob
   

   Note

   'az storage account create' may fail if your Azure instance hasn't enabled the "Microsoft.Storage" service. If you hit a failure register the Microsoft.Storage resource provider in your subscription.

2. Generate a discovery document. Upload it the storage account, and then verify that it's publicly accessible.

   cat <<EOF > openid-configuration.json
   {
     "issuer": "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}/",
     "jwks_uri": "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}/openid/v1/jwks",
     "response_types_supported": [
       "id_token"
     ],
     "subject_types_supported": [
       "public"
     ],
     "id_token_signing_alg_values_supported": [
       "RS256"
     ]
   }
   EOF
   

   az storage blob upload --container-name "${AZURE_STORAGE_CONTAINER}" --file openid-configuration.json --name .well-known/openid-configuration
   curl -s "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}/.well-known/openid-configuration"
   

3. Obtain the public key for your cluster's service account issuer from its private key. You'll likely need to run this command as a superuser. The following example is for k3s. Your cluster may store the service account issuer private key at a different location.

   sudo openssl rsa -in /var/lib/rancher/k3s/server/tls/service.key -pubout -out sa.pub
   

4. Download the latest azwi command line tool, which you can use to create a JWKS document from the public key, and untar it:

   tar -xzf <path to downloaded azwi tar.gz>
   

5. Generate the JWKS document. Upload it to the storage account, and then verify that it's publicly accessible.

   ./azwi jwks --public-keys sa.pub --output-file jwks.json
   az storage blob upload --container-name "${AZURE_STORAGE_CONTAINER}" --file jwks.json --name openid/v1/jwks
   curl -s "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_STORAGE_CONTAINER}/openid/v1/jwks"
   

Configure your cluster's Service Account token issuer with the hosted URL

Your cluster must also be configured to issue Service Account tokens with an issuer URL (service-account-issuer) field that points to the storage account you created in the previous section. For more background on the federation configuration, see cluster configuration for an OIDC issuer.

Optionally, you can configure limits on the Secret Store's own permissions as a privileged resource running in the control plane by configuring OwnerReferencesPermissionEnforcement admission controller. This admission controller constrains how much the Secret Store can change other objects in the cluster.

Your Kubernetes cluster must be running Kubernetes version 1.27 or higher.

1. Configure your kube-apiserver with the issuer URL field and permissions enforcement. The following example is for a k3s cluster. Your cluster may have different means for changing API server arguments: --kube-apiserver-arg="--service-account-issuer=${SERVICE_ACCOUNT_ISSUER}" and --kube-apiserver-arg="--enable-admission-plugins=OwnerReferencesPermissionEnforcement".

   • Get the service account issuer url.

     echo $SERVICE_ACCOUNT_ISSUER
     

   • Open the K3s server configuration file.

     sudo nano /etc/systemd/system/k3s.service
     

   • Edit the server configuration to look like the following example, replacing <SERVICE_ACCOUNT_ISSUER> with the output from echo $SERVICE_ACCOUNT_ISSUER.

     ExecStart=/usr/local/bin/k3s \
       server --write-kubeconfig-mode=644 \
          --kube-apiserver-arg="--service-account-issuer=<SERVICE_ACCOUNT_ISSUER>" \
          --kube-apiserver-arg="--enable-admission-plugins=OwnerReferencesPermissionEnforcement"
     

2. Restart your kube-apiserver.

   sudo systemctl daemon-reload
   sudo systemctl restart k3s
   

Create an Azure Key Vault

Create an Azure Key Vault and add a secret. If you already have an Azure Key Vault and secret, you can skip this section.

1. Create an Azure Key Vault:

   az keyvault create --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --name "${KEYVAULT_NAME}" --enable-rbac-authorization
   

2. Give yourself 'Secrets Officer' permissions on the vault, so you can create a secret:

   az role assignment create --role "Key Vault Secrets Officer" --assignee ${CURRENT_USER} --scope /subscriptions/${SUBSCRIPTION}/resourcegroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEYVAULT_NAME}
   

3. Create a secret and update it so you have two versions:

   az keyvault secret set --vault-name "${KEYVAULT_NAME}" --name "${KEYVAULT_SECRET_NAME}" --value 'Hello!'
   az keyvault secret set --vault-name "${KEYVAULT_NAME}" --name "${KEYVAULT_SECRET_NAME}" --value 'Hello2'
   

Create a user-assigned managed identity

Next, create a user-assigned managed identity and give it permissions to access the Azure Key Vault. If you already have a managed identity with Key Vault Reader and Key Vault Secrets User permissions to the Azure Key Vault, you can skip this section. For more information, see Create a user-assigned managed identities and Using Azure RBAC secret, key, and certificate permissions with Key Vault.

1. Create the user-assigned managed identity:

   az identity create --name "${USER_ASSIGNED_IDENTITY_NAME}" --resource-group "${RESOURCE_GROUP}" --location "${LOCATION}" --subscription "${SUBSCRIPTION}"
   

2. Give the identity Key Vault Reader and Key Vault Secrets User permissions. You may need to wait a moment for replication of the identity creation before these commands succeed:

   export USER_ASSIGNED_CLIENT_ID="$(az identity show --resource-group "${RESOURCE_GROUP}" --name "${USER_ASSIGNED_IDENTITY_NAME}" --query 'clientId' -otsv)"
   az role assignment create --role "Key Vault Reader" --assignee "${USER_ASSIGNED_CLIENT_ID}" --scope /subscriptions/${SUBSCRIPTION}/resourcegroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEYVAULT_NAME}
   az role assignment create --role "Key Vault Secrets User" --assignee "${USER_ASSIGNED_CLIENT_ID}" --scope /subscriptions/${SUBSCRIPTION}/resourcegroups/${RESOURCE_GROUP}/providers/Microsoft.KeyVault/vaults/${KEYVAULT_NAME}
   

Create a federated identity credential

Create a Kubernetes service account for the workload that needs access to secrets. Then, create a federated identity credential to link between the managed identity, the OIDC service account issuer, and the Kubernetes Service Account.

1. Create a Kubernetes Service Account that will be federated to the managed identity. Annotate it with details of the associated user-assigned managed identity.

   kubectl create ns ${KUBERNETES_NAMESPACE}
   

   cat <<EOF | kubectl apply -f -
     apiVersion: v1
     kind: ServiceAccount
     metadata:
       name: ${SERVICE_ACCOUNT_NAME}
       namespace: ${KUBERNETES_NAMESPACE}
   EOF
   

2. Create a federated identity credential:

   az identity federated-credential create --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} --identity-name ${USER_ASSIGNED_IDENTITY_NAME} --resource-group ${RESOURCE_GROUP} --issuer ${SERVICE_ACCOUNT_ISSUER} --subject system:serviceaccount:${KUBERNETES_NAMESPACE}:${SERVICE_ACCOUNT_NAME}
   

Install and use the Secret Store

The Secret Store is available as an Azure Arc extension. An Azure Arc-enabled Kubernetes cluster can be extended with Azure Arc-enabled Kubernetes extensions. Extensions enable Azure capabilities on your connected cluster and provide an Azure Resource Manager-driven experience for the extension installation and lifecycle management.

The Secret Store is installed as an Azure Arc extension

Install cert-manager and trust-manager

cert-manager and trust-manager are required for secure communication of logs between cluster services and must be installed before the Arc extension.

1. Install cert-manager.

   kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.3/cert-manager.yaml
   

2. Install trust-manager.

   helm repo add jetstack https://charts.jetstack.io
   helm repo update
   helm upgrade trust-manager jetstack/trust-manager --install --namespace cert-manager --wait
   

Install the Secret Store Azure Arc extension

Be sure that your Kubernetes cluster is connected to Azure Arc before installing the extension.

1. Set two environment variables for the resource group and name of your connected cluster. If you followed the quickstart linked earlier, these are 'AzureArcTest' and 'AzureArcTest1' respectively.

   export ARC_RESOURCE_GROUP="AzureArcTest"
   export ARC_CLUSTER_NAME="AzureArcTest1"
   

2. Install the Secret Store extension to your Arc-enabled cluster using the following command:

   az k8s-extension create \
     --cluster-name ${ARC_CLUSTER_NAME} \
     --cluster-type connectedClusters \
     --extension-type microsoft.azure.secretstore \
     --resource-group ${ARC_RESOURCE_GROUP} \
     --release-train preview \
     --name ssarcextension \
     --scope cluster 
   

   If desired, you can optionally modify the default rotation poll interval by adding --configuration-settings rotationPollIntervalInSeconds=<time_in_seconds>:

   Parameter name	Description	Default value
   rotationPollIntervalInSeconds	Specifies how quickly the Secret Store checks or updates the secret it's managing.	3600 (1 hour)

Configure the Secret Store

Configure the installed extension with information about your Azure Key Vault and which secrets to synchronize to your cluster by defining instances of Kubernetes custom resources. You create two types of custom resources:

• A SecretProviderClass object to define the connection to the Key Vault.
• A SecretSync object for each secret to be synchronized.

Create a SecretProviderClass resource

The SecretProviderClass resource is used to define the connection to the Azure Key Vault, the identity to use to access the vault, which secrets to synchronize, and the number of versions of each secret to retain locally.

You need a separate SecretProviderClass for each Azure Key Vault you intend to synchronize, for each identity used for access to an Azure Key Vault, and for each target Kubernetes namespace.

Create one or more SecretProviderClass YAML files with the appropriate values for your Key Vault and secrets by following this example.

cat <<EOF > spc.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: secret-provider-class-name                      # Name of the class; must be unique per Kubernetes namespace
  namespace: ${KUBERNETES_NAMESPACE}                    # Kubernetes namespace to make the secrets accessible in
spec:
  provider: azure
  parameters:
    clientID: "${USER_ASSIGNED_CLIENT_ID}"               # Managed Identity Client ID for accessing the Azure Key Vault with.
    keyvaultName: ${KEYVAULT_NAME}                       # The name of the Azure Key Vault to synchronize secrets from.
    objects: |
      array:
        - |
          objectName: ${KEYVAULT_SECRET_NAME}            # The name of the secret to sychronize.
          objectType: secret
          objectVersionHistory: 2                       # [optional] The number of versions to synchronize, starting from latest.
    tenantID: "${AZURE_TENANT_ID}"                       # The tenant ID of the Key Vault 
EOF

Create a SecretSync object

Each synchronized secret also requires a SecretSync object, to define cluster-specific information. Here you specify information such as the name of the secret in your cluster and names for each version of the secret stored in your cluster.

Create one SecretSync object YAML file for each secret, following this template. The Kubernetes namespace should match the namespace of the matching SecretProviderClass.

cat <<EOF > ss.yaml
apiVersion: secret-sync.x-k8s.io/v1alpha1
kind: SecretSync
metadata:
  name: secret-sync-name                                  # Name of the object; must be unique per Kubernetes namespace
  namespace: ${KUBERNETES_NAMESPACE}                      # Kubernetes namespace
spec:
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}             # The Kubernetes service account to be given permissions to access the secret.
  secretProviderClassName: secret-provider-class-name     # The name of the matching SecretProviderClass with the configuration to access the AKV storing this secret
  secretObject:
    type: Opaque
    data:
    - sourcePath: ${KEYVAULT_SECRET_NAME}/0                # Name of the secret in Azure Key Vault with an optional version number (defaults to latest)
      targetKey: ${KEYVAULT_SECRET_NAME}-data-key0         # Target name of the secret in the Kubernetes Secret Store (must be unique)
    - sourcePath: ${KEYVAULT_SECRET_NAME}/1                # [optional] Next version of the AKV secret. Note that versions of the secret must match the configured objectVersionHistory in the secrets provider class 
      targetKey: ${KEYVAULT_SECRET_NAME}-data-key1         # [optional] Next target name of the secret in the K8s Secret Store
EOF

Apply the configuration CRs

Apply the configuration custom resources (CRs) using the kubectl apply command:

kubectl apply -f ./spc.yaml
kubectl apply -f ./ss.yaml

The Secret Store automatically looks for the secrets and begins syncing them to the cluster.

View configuration options

To view additional configuration options for these two custom resource types, use the kubectl describe command to inspect the CRDs in the cluster:

# Get the name of any applied CRD(s)
kubectl get crds -o custom-columns=NAME:.metadata.name

# View the full configuration options and field parameters for a given CRD
kubectl describe crd secretproviderclass
kubectl describe crd secretsync

Observe secrets synchronizing to the cluster

Once the configuration is applied, secrets begin syncing to the cluster automatically at the cadence specified when installing the Secret Store.

View synchronized secrets

View the secrets synchronized to the cluster by running the following command:

# View a list of all secrets in the namespace
kubectl get secrets -n ${KUBERNETES_NAMESPACE}

# View details of all secrets in the namespace
kubectl get secrets -n ${KUBERNETES_NAMESPACE} -o yaml

View last sync status

To view the status of the most recent synchronization for a given secret, use the kubectl describe command for the SecretSync object. The output includes the secret creation timestamp, the versions of the secret, and detailed status messages for each synchronization event. This output can be used to diagnose connection or configuration errors, and to observe when the secret value changes.

kubectl describe secretsync secret-sync-name -n ${KUBERNETES_NAMESPACE}

View secrets values

To view the synchronized secret values, now stored in the Kubernetes secret store, use the following command:

kubectl get secret secret-sync-name -n ${KUBERNETES_NAMESPACE} -o jsonpath="{.data.${KEYVAULT_SECRET_NAME}-data-key0}" | base64 -d
kubectl get secret secret-sync-name -n ${KUBERNETES_NAMESPACE} -o jsonpath="{.data.${KEYVAULT_SECRET_NAME}-data-key1}" | base64 -d

Troubleshooting

The Secret Store is a Kubernetes deployment that contains a pod with two containers: the controller, which manages storing secrets in the cluster, and the provider, which manages access to, and pulling secrets from, the Azure Key Vault. Each synchronized secret has a SecretSync object that contains the status of the synchronization of that secret from Azure Key Vault to the cluster secret store.

To troubleshoot an issue, start by looking at the state of the SecretSync object, as described in View last sync status. The following table lists common status types, their meanings, and potential troubleshooting steps to resolve errors.

Remove the Secret Store

To remove the Secret Store and stop synchronizing secrets, uninstall it with the az k8s-extension delete command:

az k8s-extension delete --name ssarcextension --cluster-name $ARC_CLUSTER_NAME  --resource-group $RESOURCE_GROUP  --cluster-type connectedClusters    

Uninstalling the extension doesn't remove secrets, SecretSync objects, or CRDs from the cluster. These objects must be removed directly with kubectl.

Deleting the SecretSync CRD removes all SecretSync objects, and by default removes all owned secrets, but secrets may persist if:

• You modified ownership of any of the secrets.
• You changed the garbage collection settings in your cluster, including setting different finalizers.

In the above cases, secrets must be deleted directly using kubectl.

Next steps

• Learn more about Azure Arc extensions.
• Learn more about Azure Key Vault.