How to use Service Connector in Azure Kubernetes Service (AKS)

  • Article

Azure Kubernetes Service (AKS) is one of the compute services supported by Service Connector. This article aims to help you understand:

  • What operations are made on the cluster when creating a service connection.
  • How to use the kubernetes resources Service Connector creates.
  • How to troubleshoot and view logs of Service Connector in an AKS cluster.

Prerequisites

What operations Service Connector makes on the cluster

Depending on the different target services and authentication types selected when creating a service connection, Service Connector makes different operations on the AKS cluster. The following lists the possible operations made by Service Connector.

Add the Service Connector kubernetes extension

A kubernetes extension named sc-extension is added to the cluster the first time a service connection is created. Later on, the extension helps create kubernetes resources in user's cluster, whenever a service connection request comes to Service Connector. You can find the extension in your AKS cluster in the Azure portal, in the Extensions + applications menu.

Screenshot of the Azure portal, view AKS extension.

The extension is also where the cluster connections metadata are stored. Uninstalling the extension makes all the connections in the cluster unavailable. The extension operator is hosted in the cluster namespace sc-system.

Create kubernetes resources

Service Connector creates some kubernetes resources to the namespace the user specified when creating a service connection. The kubernetes resources store the connection information, which is needed by the user's workload definitions or application code to talk to target services. Depending on different authentication types, different kubernetes resources are created. For the Connection String and Service Principal auth types, a kubernetes secret is created. For the Workload Identity auth type, a kubernetes service account is also created in addition to a kubernetes secret.

You can find the kubernetes resources created by Service Connector for each service connection on the Azure portal in your kubernetes resource, in the Service Connector menu.

Screenshot of the Azure portal, view Service Connector created kubernetes resources.

Deleting a service connection doesn't delete the associated Kubernetes resource. If necessary, remove your resource manually, using for example the kubectl delete command.

Enable the azureKeyvaultSecretsProvider addon

If target service is Azure Key Vault and the Secret Store CSI Driver is enabled when creating a service connection, Service Connector enables the azureKeyvaultSecretsProvider add-on for the cluster.

Screenshot of the Azure portal, enabling CSI driver for keyvault when creating a connection.

Follow the Connect to Azure Key Vault using CSI driver tutorialto set up a connection to Azure Key Vault using Secret Store CSI driver.

Enable workload identity and OpenID Connect (OIDC) issuer

If the authentication type is Workload Identity when creating a service connection, Service Connector enables workload identity and OIDC issuer for the cluster.

Screenshot of the Azure portal, using workload identity to create a connection.

When the authentication type is Workload Identity, a user-assigned managed identity is needed to create the federated identity credential. Learn more from what are workload identities, or follow the tutorialto set up a connection to Azure Storage using workload identity.

How to use the Service Connector created kubernetes resources

Different kubernetes resources are created when the target service type and authentication type are different. The following sections show how to use the Service Connector created kubernetes resources in your cluster workloads definition and application codes.

Kubernetes secret

A kubernetes secret is created when the authentication type is Connection String or Service Principal. Your cluster workload definition can reference the secret directly. The following snnipet is an example.

apiVersion: batch/v1
kind: Job
metadata:
  namespace: default
  name: sc-sample-job
spec:
  template:
    spec:
      containers:
      - name: raw-linux
        image: alpine
        command: ['printenv']
        envFrom:
          - secretRef:
              name: <SecretCreatedByServiceConnector>
      restartPolicy: OnFailure

Then, your application codes can consume the connection string in the secret from environment variable. You can check the sample code to learn more about the environment variable names and how to use them in your application codes to authenticate to different target services.

Kubernetes service account

Both a kubernetes service account and a secret are created when the authentication type is Workload Identity. Your cluster workload definition can reference the service account and secret to authenticate through workload identity. The following snipet provides an example.

apiVersion: batch/v1
kind: Job
metadata:
  namespace: default
  name: sc-sample-job
  labels:
    azure.workload.identity/use: "true"
spec:
  template:
    spec:
      serviceAccountName: <ServiceAccountCreatedByServiceConnector>
      containers:
      - name: raw-linux
        image: alpine
        command: ['printenv']
        envFrom:
          - secretRef:
              name: <SecretCreatedByServiceConnector>
      restartPolicy: OnFailure

You may check the tutorial to learn how to connect to Azure Storage using workload identity.

How to troubleshoot and view logs

If an error happens and couldn't be mitigated by retrying when creating a service connection, the following methods can help gather more information for troubleshooting.

Check Service Connector kubernetes extension

Service Connector kubernetes extension is built on top of Azure Arc-enabled Kubernetes cluster extensions. Use the following commands to investigate if there are any errors during the extension installation or updating.

  1. Install the k8s-extension Azure CLI extension.
az extension add --name k8s-extension
  1. Get the Service Connector extension status. Check the statuses property in the command output to see if there are any errors.
az k8s-extension show \
    --resource-group MyClusterResourceGroup \
    --cluster-name MyCluster \
    --cluster-type managedClusters \
    --name sc-extension

Check kubernetes cluster logs

If there's an error during the extension installation, and the error message in the statuses property doesn't provide enough information about what happened, you can further check the kubernetes logs with the followings steps.

  1. Connect to your AKS cluster.

    az aks get-credentials \
    --resource-group MyClusterResourceGroup \
    --name MyCluster

  2. Service Connector extension is installed in the namespace sc-system through helm chart, check the namespace and the helm release by following commands.

    • Check the namespace exists.
    kubectl get ns
    • Check the helm release status.
    helm list -n sc-system

  3. During the extension installation or updating, a kubernetes job called sc-job creates the kubernetes resources for the service connection. The job execution failure usually causes the extension failure. Check the job status by running the following commands. If sc-job doesn't exist in sc-system namespace, it should have been executed successfully. This job is designed to be automatically deleted after successful execution.

    • Check the job exists.
    kubectl get job -n sc-system
    • Get the job status.
    kubectl describe job/sc-job -n sc-system
    • View the job logs.
    kubectl logs job/sc-job -n sc-system

Common errors and mitigations

Conflict

Error Message: Operation returned an invalid status code: Conflict.

Reason: This error usually occurs when attempting to create a service connection while the AKS (Azure Kubernetes Service) cluster is in an updating state. The service connection update conflicts with the ongoing update. It could also happen when your subscription is not resgitered for the Microsoft.KubernetesConfiguration resource provider.

Mitigation:

  • Run the following command to make sure your subscription is registered for Microsoft.KubernetesConfiguration resource provider.

    az provider register -n Microsoft.KubernetesConfiguration

  • Ensure your cluster is in a "Succeeded" state and retry the creation.

Timeout

Error Message:

  • Long running operation failed with status 'Failed'. Unable to get a response from the Agent in time.
  • Timed out waiting for the resource to come to a ready/completed state

Reason: This error often happens when the Kubernetes job used to create or update the Service Connector cluster extension fails to be scheduled due to resource limitations or other issues.

Mitigation: Refer to Check Kubernetes cluster logs to identify and resolve the detailed reasons. A common issue is that no nodes are available due to preemption. In this case, consider adding more nodes or enabling auto-scaling for your nodes.

Unauthorized resource access

Error Message: You do not have permission to perform ... If access was recently granted, please refresh your credentials.

Reason: Service Connector requires permissions to operate the Azure resources you want to connect to, in order to perform connection operations on your behalf. This error indicates a lack of necessary permissions on some Azure resources.

Mitigation: Check the permissions on the Azure resources specified in the error message. Obtain the required permissions and retry the creation.

Missing subscription registration

Error Message: The subscription is not registered to use namespace 'Microsoft.KubernetesConfiguration'

Reason: Service Connector requires the subscription to be registered for Microsoft.KubernetesConfiguration, which is the resource provider for Azure Arc-enabled Kubernetes cluster extensions.

Mitigation: Register the Microsoft.KubernetesConfiguration resource provider by running the following command. For more information on resource provider registration errors, please refer to this tutorial.

az provider register -n Microsoft.KubernetesConfiguration

Other issues

If the above mitigations don't resolve your issue, try resetting the service connector cluster extension by removing it and then retrying the creation. This method is expected to resolve most issues related to the Service Connector cluster extension.

Use the following CLI commands to reset the extension:

az extension add --name k8s-extension

az k8s-extension delete \
    --resource-group <MyClusterResourceGroup> \
    --cluster-name <MyCluster> \
    --cluster-type managedClusters \
    --name sc-extension

Next steps

Learn how to integrate different target services and read about their configuration settings and authentication methods.

Learn about how to integrate storage blob