Tutorial: Connect to Azure OpenAI Service in AKS using Workload Identity (preview)

In this tutorial, you learn how to create a pod in an Azure Kubernetes (AKS) cluster, which talks to Azure OpenAI Service using workload identity and Service Connector. In this tutorial, you complete the following tasks:

Prerequisites

• An Azure account with an active subscription. Create an account for free.

• • Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.

    

  • If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.

    • If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.

    • When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.

    • Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.

• Install Docker and kubectl to manage container image and Kubernetes resources.
• A basic understanding of container and AKS. Get started from preparing an application for AKS.
• A basic understanding of workload identity.
• Access permissions to create Azure OpenAI resources and to deploy models.

Create Azure resources

You start this tutorial by creating several Azure resources.

1. Create a resource group for this tutorial.

   az group create \
       --name MyResourceGroup \
       --location eastus
   

2. Create an AKS cluster with the following command, or by referring to the AKS quickstart. In this tutorial, we create the service connection and pod definition and deploy the sample application to this cluster.

   az aks create \
       --resource-group MyResourceGroup \
       --name MyAKSCluster \
       --enable-managed-identity \
       --node-count 1
       --generate-ssh-keys
   

3. Connect to the cluster using the az aks get-credentials command.

   az aks get-credentials \
       --resource-group MyResourceGroup \
       --name MyAKSCluster
   

4. Create an Azure OpenAI Service resource using the az cognitiveservices account create command. Optionally refer to this tutorial for more instructions. Azure OpenAI Service is the target service that we'll connect to the AKS cluster.

   az cognitiveservices account create \
       --resource-group MyResourceGroup \
       --name MyOpenAIService \
       --location eastus \
       --kind OpenAI \
       --sku s0 \
       --custom-domain myopenaiservice \
       --subscription <SubscriptionID>
   

5. Deploy a model with the az cognitiveservices deployment create command. The model is used in the sample application to test the connection.

   az cognitiveservices account deployment create \
       --resource-group MyResourceGroup \
       --name MyOpenAIService
       --deployment-name MyModel \
       --model-name gpt-4 \
       --model-version 0613 \
       --model-format OpenAI \
       --sku-name "Standard"
       --capacity 1
   

6. Create an Azure Container Registry (ACR) resource with the az acr create command, or referring to this tutorial. The registry hosts the container image of the sample application, which the AKS pod definition consumes.

   az acr create \
       --resource-group MyResourceGroup \
       --name myregistry \
       --sku Standard
   

7. Enable anonymous pull using az acr update command so that the AKS cluster can consume the images in the registry.

   az acr update \
       --resource-group MyResourceGroup \
       --name MyRegistry \
       --anonymous-pull-enabled
   

8. Create a user-assigned managed identity with the az identity create command, or by referring to this tutorial. When the connection is created, the user-assigned managed identity is used to enable the workload identity for AKS workloads.

   az identity create \
       --resource-group MyResourceGroup \
       --name MyIdentity
   

Create a service connection in AKS with Service Connector (preview)

Create a service connection between an AKS cluster and Azure OpenAI Service in the Azure portal or the Azure CLI.

az aks connection create cognitiveservices \
   --workload-identity <user-identity-resource-id>

Clone sample application

1. Clone the sample repository:

   git clone https://github.com/Azure-Samples/serviceconnector-aks-samples.git
   

2. Go to the repository's sample folder for Azure OpenAI:

   cd serviceconnector-aks-samples/azure-openai-workload-identity
   

3. Replace the <MyModel> placeholder in the app.py file with the model name we deployed.

Build and push container images

1. Build and push the images to your ACR using the Azure CLI az acr build command.

   az acr build --registry <MyRegistry> --image sc-demo-openai-identity:latest ./
   

2. View the images in your ACR instance using the az acr repository list command.

   az acr repository list --name <MyRegistry> --output table
   

Run application and test connection

1. Replace the placeholders in the pod.yaml file in the azure-openai-workload-identity folder.

   • Replace <YourContainerImage> with the name of the image we built earlier. For example, <MyRegistry>.azurecr.io/sc-demo-openai-identity:latest.
   • Replace <ServiceAccountCreatedByServiceConnector> with the service account created by Service Connector after the connection creation. You may check the service account name in the Azure portal, in the Service Connector pane.
   • Replace <SecretCreatedByServiceConnector> with the secret created by Service Connector after the connection creation. You may check the secret name in the Azure portal, in the Service Connector pane.

2. Deploy the pod to your cluster with the kubectl apply command, which creates a pod named sc-demo-openai-identity in the default namespace of your AKS cluster. Install kubectl locally using the az aks install-cli command if it isn't installed.

   kubectl apply -f pod.yaml
   

3. Check if the deployment was successful by viewing the pod with kubectl.

   kubectl get pod/sc-demo-openai-identity
   

4. Check that connection is established by viewing the logs with kubectl.

   kubectl logs pod/sc-demo-openai-identity
   

Clean up resources

If you no longer need the resources created in this tutorial, clean them up by deleting the resource group.

az group delete \
    --resource-group MyResourceGroup

Next steps

Read the following articles to learn more about Service Connector concepts and how it helps AKS connect to services.