Istio service mesh add-on plug-in CA certificate troubleshooting

This article discusses common troubleshooting issues with the Istio add-on plug-in certificate authority (CA) certificates feature, and it offers solutions to fix these issues. The article also reviews the general process of setting up plug-in CA certificates for the service mesh add-on.

Prerequisites

• Azure CLI.

• The Kubernetes kubectl tool, or a similar tool, to connect to the cluster. To install kubectl by using Azure CLI, run the az aks install-cli command.

• The following Linux-style standard shell tools:

  • grep
  • sort
  • tail
  • awk
  • xargs

• The jq tool for querying JSON data.

General setup process

• Before you enable the Istio add-on to use the plug-in CA certificates feature, you have to enable the Azure Key Vault provider for Secrets Store add-on on the cluster. Make sure that the Azure Key Vault and the cluster are on the same Azure tenant.

• After the Azure Key Vault secrets provider add-on is enabled, you have to set up access to the Azure Key Vault for the user-assigned managed identity that the add-on creates.

• After you grant permission for the user-assigned managed identity to access the Azure Key Vault, you can use the plug-in CA certificates feature together with the Istio add-on. For more information, see the Enable the Istio add-on to use a plug-in CA certificate section.

• For the cluster to auto-detect changes in the Azure Key Vault secrets, you have to enable auto-rotation for the Azure Key Vault secrets provider add-on.

• Although changes to the intermediate certificate are applied automatically, changes to the root certificate are only picked up by the control plane after the istiod deployment is restarted by a cronjob that the add-on deploys, as explained in the Deployed resources section. This cronjob runs at a 10-minute interval.

Enable the Istio add-on to use a plug-in CA certificate

The Istio add-on plug-in CA certificates feature allows you to configure plug-in root and intermediate certificates for the mesh. To provide plug-in certificate information when you enable the add-on, specify the following parameters for the az aks mesh enable command in Azure CLI.

If you want to use the plug-in CA certificates feature, you must specify all five parameters. All Azure Key Vault objects are expected to be of the type Secret.

For more information, see Plug in CA certificates for Istio-based service mesh add-on on Azure Kubernetes Service.

Deployed resources

As part of the add-on deployment for the plug-in certificates feature, the following resources are deployed onto the cluster:

• The cacerts Kubernetes secret is created in the aks-istio-system namespace at the time of the add-on deployment. This secret contains synchronized Azure Key Vault secrets:

  kubectl describe secret cacerts --namespace aks-istio-system
  

  Name:         cacerts
  Namespace:    aks-istio-system
  Labels:       secrets-store.csi.k8s.io/managed=true
  Annotations:  <none>
  
  Type:  opaque
  
  Data
  ====
  ca-cert.pem:     1968 bytes
  ca-key.pem:      3272 bytes
  cert-chain.pem:  3786 bytes
  root-cert.pem:   3636 bytes
  

• The istio-spc-asm-1-21 SecretProviderClass object is created in the aks-istio-system namespace at the time of the add-on deployment. This resource contains Azure-specific parameters for the Secrets Store Container Storage Interface (CSI) driver:

  kubectl get secretproviderclass --namespace aks-istio-system
  

  NAME                 AGE
  istio-spc-asm-1-21   14h
  

• The istio-ca-root-cert configmap is created in the aks-istio-system namespace and all user-managed namespaces. This configmap contains the root certificate that the certificate authority uses, and it's used by workloads in the namespaces to validate workload-to-workload communication, as follows:

  kubectl describe configmap istio-ca-root-cert --namespace aks-istio-system
  

  Name:         istio-ca-root-cert
  Namespace:    aks-istio-system
  Labels:       istio.io/config=true
  Annotations:  <none>
  
  Data
  ====
  root-cert.pem:
  ----
  -----BEGIN CERTIFICATE-----
  <certificate data>
  -----END CERTIFICATE-----
  

• The istio-cert-validator-cronjob-asm-1-21 cronjob object is created in the aks-istio-system namespace. This cronjob is scheduled to run every 10 minutes to check for updates on the root certificate. If the root certificate that's in the cacerts Kubernetes secret doesn't match the istio-ca-root-cert configmap in the aks-istio-system namespace, it restarts the istiod-asm-1-21 deployment:

  kubectl get cronjob --namespace aks-istio-system
  

  NAME                                    SCHEDULE       SUSPEND   ACTIVE
  istio-cert-validator-cronjob-asm-1-21   */10 * * * *   False     0     
  

  You can run the following command to check the cronjob logs for the last run:

  kubectl logs --namespace aks-istio-system $(kubectl get pods --namespace aks-istio-system | grep 'istio-cert-validator-cronjob-' | sort -k8 | tail -n 1 | awk '{print $1}')
  

  This command generates one of the following output messages, depending on whether a root certificate update was detected:

  Root certificate update not detected.
  

  Root certificate update detected. Restarting deployment...
  deployment.apps/istiod-asm-1-21 restarted
  Deployment istiod-asm-1-21 restarted.
  

Determine certificate type in deployment logs

You can view the istiod deployment logs to determine whether you have a self-signed CA certificate or a plug-in CA certificate. To view the logs, run the following command:

kubectl logs deploy/istiod-asm-1-21 --container discovery --namespace aks-istio-system | grep -v validationController

Immediately before each certificate log entry is another log entry that describes that kind of certificate. For a self-signed CA certificate, the entry states "No plugged-in cert at etc/cacerts/ca-key.pem; self-signed cert is used." For a plug-in certificate, the entry states "Use plugged-in cert at etc/cacerts/ca-key.pem." Sample log entries that pertain to the certificates are shown in the following tables.

• Log entries for a self-signed CA certificate

  Timestamp	Log level	Message
  2023-11-20T23:27:36.649019Z	info	Using istiod file format for signing ca files
  2023-11-20T23:27:36.649032Z	info	No plugged-in cert at etc/cacerts/ca-key.pem; self-signed cert is used
  2023-11-20T23:27:36.649536Z	info	x509 cert - <certificate-details>
  2023-11-20T23:27:36.649552Z	info	Istiod certificates are reloaded
  2023-11-20T23:27:36.649613Z	info	spiffe Added 1 certs to trust domain cluster.local in peer cert verifier

• Log entries for a plug-in CA certificate

  Timestamp	Log level	Message
  2023-11-21T00:20:25.808396Z	info	Using istiod file format for signing ca files
  2023-11-21T00:20:25.808412Z	info	Use plugged-in cert at etc/cacerts/ca-key.pem
  2023-11-21T00:20:25.808731Z	info	x509 cert - <certificate-details>
  2023-11-21T00:20:25.808764Z	info	x509 cert - <certificate-details>
  2023-11-21T00:20:25.808799Z	info	x509 cert - <certificate-details>
  2023-11-21T00:20:25.808803Z	info	Istiod certificates are reloaded
  2023-11-21T00:20:25.808873Z	info	spiffe Added 1 certs to trust domain cluster.local in peer cert verifier

The certificate details in a log entry are shown as comma-separated values for the issuer, subject, serial number (SN—a long hexadecimal string), and the beginning and ending timestamp values that define when the certificate is valid.

For a self-signed CA certificate, there's one detail entry. Sample values for this certificate are shown in the following table.

For a plug-in CA certificate, there are three detail entries. The other two entries are for a root certificate update and a change to the intermediate certificate. Sample values for these entries are shown in the following table.

Troubleshoot common issues

Issue 1: Access to Azure Key Vault is set up incorrectly

After you enable the Azure Key Vault secrets provider add-on, you have to grant access for the user-assigned managed identity of the add-on to the Azure Key Vault. Setting up access to Azure Key Vault incorrectly causes the add-on installation to stall.

kubectl get pods --namespace aks-istio-system

In the list of pods, you can see that the istiod-asm-1-21 pods are stuck in an Init:0/2 state.

To verify the Azure Key Vault access issue, run the kubectl get pods command to locate pods that have the secrets-store-provider-azure label in the kube-system namespace:

kubectl get pods --selector app=secrets-store-provider-azure --namespace kube-system --output name | xargs -I {} kubectl logs --namespace kube-system {}

The following sample output shows that a "403 Forbidden" error occurred because you don't have "get" permissions for secrets on the Key Vault:

"failed to process mount request" err="failed to get objectType:secret, objectName:<secret-object-name>, objectVersion:: keyvault.BaseClient#GetSecret: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code=\"Forbidden\" Message=\"The user, group or application 'appid=<appid>;oid=<oid>;iss=<iss>' does not have secrets get permission on key vault 'MyAzureKeyVault;location=eastus'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287\" InnerError={\"code\":\"AccessDenied\"}"

To fix this problem, set up access to the user-assigned managed identity for the Azure Key Vault secrets provider add-on by obtaining Get and List permissions on Azure Key Vault secrets and reinstalling the Istio add-on. First, get the object ID of the user-assigned managed identity for the Azure Key Vault secrets provider add-on by running the az aks show command:

OBJECT_ID=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER --query 'addonProfiles.azureKeyvaultSecretsProvider.identity.objectId')

To set the access policy, run the following az keyvault set-policy command by specifying the object ID that you obtained:

az keyvault set-policy --name $AKV_NAME --object-id $OBJECT_ID --secret-permissions get list

Issue 2: Auto-detection of Key Vault secret changes isn't set up

For a cluster to auto-detect changes in the Azure Key Vault secrets, you have to enable auto-rotation for the Azure Key Vault provider add-on. Auto-rotation can detect changes in intermediate and root certificates automatically. For a cluster that enables the Azure Key Vault provider add-on, run the following az aks show command to check whether auto-rotation is enabled:

az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER | jq -r '.addonProfiles.azureKeyvaultSecretsProvider.config.enableSecretRotation'

If the cluster enabled the Azure Key Vault provider add-on, run the following az aks show command to determine the rotation poll interval:

az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER | jq -r '.addonProfiles.azureKeyvaultSecretsProvider.config.rotationPollInterval'

Azure Key Vault secrets are synchronized with the cluster when the poll interval time elapses after the previous synchronization. The default interval value is two minutes.

Issue 3: Certificate values are missing or are configured incorrectly

If secret objects are missing from Azure Key Vault, or if these objects are configured incorrectly, the istiod-asm-1-21 pods might get stuck in an Init:0/2 status, delaying the installation of the add-on. To find the underlying cause of this problem, run the following kubectl describe command against the istiod deployment and view the output:

kubectl describe deploy/istiod-asm-1-21 --namespace aks-istio-system

The command displays events that might resemble the following output table. In this example, a missing secret is the cause of the problem.

Resources

• General Istio service mesh add-on troubleshooting

• Istio service mesh add-on MeshConfig troubleshooting

• Istio service mesh add-on ingress gateway troubleshooting

• Istio service mesh add-on minor revision upgrade troubleshooting

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Third-party contact disclaimer

Microsoft provides third-party contact information to help you find additional information about this topic. This contact information may change without notice. Microsoft does not guarantee the accuracy of third-party contact information.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.