Tutorial: Connect to Azure OpenAI Service in AKS using Workload Identity (preview)

Article
05/21/2024

In this tutorial, you learn how to create a pod in an Azure Kubernetes (AKS) cluster, which talks to Azure OpenAI Service using workload identity and Service Connector. In this tutorial, you complete the following tasks:

Create an AKS cluster and Azure OpenAI Service with gpt-4 model deployment.
Create a connection between the AKS cluster and Azure OpenAI with Service Connector.
Clone a sample application that will talk to the Azure OpenAI service from an AKS cluster.
Deploy the application to a pod in AKS cluster and test the connection.
Clean up resources.

Prerequisites

An Azure account with an active subscription. Create an account for free.
- Use the Bash environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
- If you prefer to run CLI reference commands locally, install the Azure CLI. If you're running on Windows or macOS, consider running Azure CLI in a Docker container. For more information, see How to run the Azure CLI in a Docker container.
  - If you're using a local installation, sign in to the Azure CLI by using the az login command. To finish the authentication process, follow the steps displayed in your terminal. For other sign-in options, see Sign in with the Azure CLI.
  - When you're prompted, install the Azure CLI extension on first use. For more information about extensions, see Use extensions with the Azure CLI.
  - Run az version to find the version and dependent libraries that are installed. To upgrade to the latest version, run az upgrade.
Install Docker and kubectl to manage container image and Kubernetes resources.
A basic understanding of container and AKS. Get started from preparing an application for AKS.
A basic understanding of workload identity.
Access permissions to create Azure OpenAI resources and to deploy models.

Create Azure resources

You start this tutorial by creating several Azure resources.

Create a resource group for this tutorial.

az group create \
    --name MyResourceGroup \
    --location eastus

Create an AKS cluster with the following command, or by referring to the AKS quickstart. In this tutorial, we create the service connection and pod definition and deploy the sample application to this cluster.
```
az aks create \
    --resource-group MyResourceGroup \
    --name MyAKSCluster \
    --enable-managed-identity \
    --node-count 1
    --generate-ssh-keys
```

Connect to the cluster using the az aks get-credentials command.

az aks get-credentials \
    --resource-group MyResourceGroup \
    --name MyAKSCluster

Create an Azure OpenAI Service resource using the az cognitiveservices account create command. Optionally refer to this tutorial for more instructions. Azure OpenAI Service is the target service that we'll connect to the AKS cluster.

az cognitiveservices account create \
    --resource-group MyResourceGroup \
    --name MyOpenAIService \
    --location eastus \
    --kind OpenAI \
    --sku s0 \
    --custom-domain myopenaiservice \
    --subscription <SubscriptionID>

Deploy a model with the az cognitiveservices deployment create command. The model is used in the sample application to test the connection.

az cognitiveservices account deployment create \
    --resource-group MyResourceGroup \
    --name MyOpenAIService
    --deployment-name MyModel \
    --model-name gpt-4 \
    --model-version 0613 \
    --model-format OpenAI \
    --sku-name "Standard"
    --capacity 1

Create an Azure Container Registry (ACR) resource with the az acr create command, or referring to this tutorial. The registry hosts the container image of the sample application, which the AKS pod definition consumes.
```
az acr create \
    --resource-group MyResourceGroup \
    --name myregistry \
    --sku Standard
```
Enable anonymous pull using az acr update command so that the AKS cluster can consume the images in the registry.
```
az acr update \
    --resource-group MyResourceGroup \
    --name MyRegistry \
    --anonymous-pull-enabled
```
Create a user-assigned managed identity with the az identity create command, or by referring to this tutorial. When the connection is created, the user-assigned managed identity is used to enable the workload identity for AKS workloads.
```
az identity create \
    --resource-group MyResourceGroup \
    --name MyIdentity
```

Create a service connection in AKS with Service Connector (preview)

Create a service connection between an AKS cluster and Azure OpenAI Service in the Azure portal or the Azure CLI.

Portal
Azure CLI

Refer to the AKS service connection quickstart for instructions to create a new connection and fill in the settings referring to the examples in the following table. Leave all other settings with their default values.

Basics tab:

Setting	Example value	Description
Kubernetes namespace	default	The Kubernetes namespace.
Service type	OpenAI Service	The target service type.
Connection name	openai_conn	Use the connection name provided by Service Connector or choose your own connection name.
Subscription	`<MySubscription>`	The subscription used for Azure OpenAI Service.
OpenAI	`<MyOpenAIService>`	The target Azure OpenAI service you want to connect to.
Client type	Python	The code language or framework you use to connect to the target service.

Authentication tab:

Authentication Setting	Example value	Description
Authentication type	Workload Identity	Service Connector authentication type.
Subscription	`<MySubscription>`	The subscription that contains the user assigned managed identity.
User assigned managed identity	`<MyIdentity>`	A user assigned managed identity is needed to enable workload identity.

Once the connection has been created, you can view its details in the Service Connector pane.

Use the Azure CLI command to create a service connection to the Azure OpenAI service, providing the following information:

Source compute service resource group name: the resource group name of the AKS cluster.
AKS cluster name: the name of your AKS cluster that connects to the target service.
Target service resource group name: the resource group name of the Azure OpenAI service.
OpenAI service name: the Azure OpenAI service that is connected.
User-assigned identity resource ID: the resource ID of the user-assigned identity used to create the workload identity.

az aks connection create cognitiveservices \
   --workload-identity <user-identity-resource-id>

Clone sample application

Clone the sample repository:

git clone https://github.com/Azure-Samples/serviceconnector-aks-samples.git

Go to the repository's sample folder for Azure OpenAI:

cd serviceconnector-aks-samples/azure-openai-workload-identity

Replace the <MyModel> placeholder in the app.py file with the model name we deployed.

Build and push container images

Build and push the images to your ACR using the Azure CLI az acr build command.

az acr build --registry <MyRegistry> --image sc-demo-openai-identity:latest ./

View the images in your ACR instance using the az acr repository list command.
```
az acr repository list --name <MyRegistry> --output table
```

Run application and test connection

Replace the placeholders in the pod.yaml file in the azure-openai-workload-identity folder.
- Replace <YourContainerImage> with the name of the image we built earlier. For example, <MyRegistry>.azurecr.io/sc-demo-openai-identity:latest.
- Replace <ServiceAccountCreatedByServiceConnector> with the service account created by Service Connector after the connection creation. You may check the service account name in the Azure portal, in the Service Connector pane.
- Replace <SecretCreatedByServiceConnector> with the secret created by Service Connector after the connection creation. You may check the secret name in the Azure portal, in the Service Connector pane.
Deploy the pod to your cluster with the kubectl apply command, which creates a pod named sc-demo-openai-identity in the default namespace of your AKS cluster. Install kubectl locally using the az aks install-cli command if it isn't installed.
```
kubectl apply -f pod.yaml
```
Check if the deployment was successful by viewing the pod with kubectl.
```
kubectl get pod/sc-demo-openai-identity
```
Check that connection is established by viewing the logs with kubectl.
```
kubectl logs pod/sc-demo-openai-identity
```

Clean up resources

If you no longer need the resources created in this tutorial, clean them up by deleting the resource group.

az group delete \
    --resource-group MyResourceGroup

Next steps

Read the following articles to learn more about Service Connector concepts and how it helps AKS connect to services.

Learn about Service Connector concepts

Use Service Connector to connect an AKS cluster to other cloud services

Share via