Send Prometheus metrics from virtual machines, scale sets, or Kubernetes clusters to an Azure Monitor workspace

Remote-write using user-assigned managed identity authentication

User-assigned managed identity authentication can be used in any Azure-managed environment. If your Prometheus service is running in a non-Azure environment, you can use Microsoft Entra ID application authentication.

To configure a user-assigned managed identity for remote-write to Azure Monitor workspace, complete the following steps.

Create a user-assigned managed identity

To create a user-managed identity to use in your remote-write configuration, see Manage user-assigned managed identities.

Note the value of the clientId of the managed identity that you created. This ID is used in the Prometheus remote write configuration.

Assign the Monitoring Metrics Publisher role to the application

On the workspace's data collection rule, assign the Monitoring Metrics Publisher role to the managed identity.

1. On the Azure Monitor workspace Overview page, select the Data collection rule link.

   

2. On the data collection rule page, select Access control (IAM).

3. Select Add, and Add role assignment.

4. Search for and select for Monitoring Metrics Publisher, and then select Next.

5. Select Managed Identity.

6. Select Select members.

7. In the Managed Entity dropdown, User-assigned managed identity.

8. Select the user-assigned identity that you want to use, then click Select.

9. Select Review + assign to complete the role assignment.

   

Assign the managed identity to a Virtual Machine or Virtual Machine Scale Set

1. In the Azure portal, go to the cluster, Virtual Machine, or Virtual Machine Scale Set's page.

2. Select Identity.

3. Select User assigned.

4. Select Add.

5. Select the user assigned managed identity that you created, then select Add.

   

Assign the managed identity for an Azure Kubernetes Service

For Azure Kubernetes services (AKS), the managed identity must be assigned to the virtual machine scale sets.

AKS creates a resource group containing the virtual machine scale sets. The resource group name is in the format MC_<resource group name>_<AKS cluster name>_<region>. For each Virtual Machine Scale Set in the resource group, assign the managed identity according to the steps in the previous section, Assign the managed identity to a Virtual Machine or Virtual Machine Scale Set.

Remote-write using Microsoft Entra ID application authentication

Microsoft Entra ID application authentication can be used in any environment. If your Prometheus service is running in an Azure-managed environment, consider using user-assigned managed identity authentication.

To configure remote-write to Azure Monitor workspace using a Microsoft Entra ID application, create a Microsoft Entra application. On Azure Monitor workspace's data collection rule, assign the Monitoring Metrics Publisher role to the Microsoft Entra application.

Create a Microsoft Entra ID application

To create a Microsoft Entra ID application using the portal, see Create a Microsoft Entra ID application and service principal that can access resources.

When you have created your Microsoft Entra application, get the client ID and generate a client secret.

1. In the list of applications, copy the value for Application (client) ID for the registered application. This value is used in the Prometheus remote write configuration as the value for client_id.

   

2. Select Certificates and Secrets

3. Select Client secrets, them select New client secret to create a new Secret

4. Enter a description, set the expiration date and select Add.

   

5. Copy the value of the secret securely. The value is used in the Prometheus remote write configuration as the value for client_secret. The client secret value is only visible when created and can't be retrieved later. If lost, you must create a new client secret.

   

Assign the Monitoring Metrics Publisher role to the application

Assign the Monitoring Metrics Publisher role on the workspace's data collection rule to the application.

1. On the Azure Monitor workspace, overview page, select the Data collection rule link.

   

2. On the data collection rule overview page, select Access control (IAM).

3. Select Add, and then select Add role assignment.

   

4. Select the Monitoring Metrics Publisher role, and then select Next.

   

5. Select User, group, or service principal, and then choose Select members. Select the application that you created, and then choose Select.

   

6. To complete the role assignment, select Review + assign.

Create user-assigned identities and Microsoft Entra ID apps using CLI

Create a user-assigned managed identity

Create a user-assigned managed identity for remote-write using the following steps:

1. Create a user-assigned managed identity
2. Assign the Monitoring Metrics Publisher role on the workspace's data collection rule to the managed identity
3. Assign the managed identity to a Virtual Machine or Virtual Machine Scale Set.

Note the value of the clientId of the managed identity that you create. This ID is used in the Prometheus remote write configuration.

1. Create a user-assigned managed identity using the following CLI command:

   az account set \
   --subscription <subscription id>
   
   az identity create \
   --name <identity name> \
   --resource-group <resource group name>
   

   The following is an example of the output displayed:

   {
     "clientId": "00001111-aaaa-2222-bbbb-3333cccc4444",
     "id": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-001/providers/Microsoft.ManagedIdentity/userAssignedIdentities/PromRemoteWriteIdentity",
     "location": "eastus",
     "name": "PromRemoteWriteIdentity",
     "principalId": "aaaaaaaa-bbbb-cccc-1111-222222222222",
     "resourceGroup": "rg-001",
     "systemData": null,
     "tags": {},
     "tenantId": "ffff5f5f-aa6a-bb7b-cc8c-dddddd9d9d9d",
     "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
   }
   

2. Assign the Monitoring Metrics Publisher role on the workspace's data collection rule to the managed identity.

   az role assignment create \
   --role "Monitoring Metrics Publisher" \
   --assignee <managed identity client ID> \
   --scope <data collection rule resource ID>
   

   For example,

   az role assignment create \
   --role "Monitoring Metrics Publisher" \
   --assignee 00001111-aaaa-2222-bbbb-3333cccc4444 \
   --scope /subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/MA_amw-001_eastus_managed/providers/Microsoft.Insights/dataCollectionRules/amw-001
   

3. Assign the managed identity to a Virtual Machine or Virtual Machine Scale Set.

   For Virtual Machines:

   az vm identity assign \
   -g <resource group name> \
   -n <virtual machine name> \
   --identities <user assigned identity resource ID>
   

   For Virtual Machine Scale Sets:

   az vmss identity assign \
   -g <resource group name> \
   -n <VSS name> \
   --identities <user assigned identity resource ID>
   

   For example, for a Virtual Machine Scale Set:

   az vm identity assign \
   -g rg-prom-on-vm \
   -n win-vm-prom \
   --identities /subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-001/providers/Microsoft.ManagedIdentity/userAssignedIdentities/PromRemoteWriteIdentity
   

For more information, see az identity create and az role assignment create.

Create a Microsoft Entra ID application

To create a Microsoft Entra ID application using CLI, and assign the Monitoring Metrics Publisher role, run the following command:

az ad sp create-for-rbac --name <application name> \
--role "Monitoring Metrics Publisher" \
--scopes <azure monitor workspace data collection rule Id>

For example,

az ad sp create-for-rbac \
--name PromRemoteWriteApp \
--role "Monitoring Metrics Publisher" \
--scopes /subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/MA_amw-001_eastus_managed/providers/Microsoft.nsights/dataCollectionRules/amw-001

The following is an example of the output displayed:

{
  "appId": "66667777-aaaa-8888-bbbb-9999cccc0000",
  "displayName": "PromRemoteWriteApp",
  "password": "Aa1Bb~2Cc3.-Dd4Ee5Ff6Gg7Hh8Ii9_Jj0Kk1Ll2",
  "tenant": "ffff5f5f-aa6a-bb7b-cc8c-dddddd9d9d9d"
}

The output contains the appId and password values. Save these values to use in the Prometheus remote write configuration as values for client_id and client_secret The password or client secret value is only visible when created and can't be retrieved later. If lost, you must create a new client secret.

For more information, see az ad app create and az ad sp create-for-rbac.

To send data to your Azure Monitor Workspace, add the following section to the configuration file (prometheus.yml) of your self-managed Prometheus instance.

remote_write:   
  - url: "<metrics ingestion endpoint for your Azure Monitor workspace>"
# AzureAD configuration.
# The Azure Cloud. Options are 'AzurePublic', 'AzureChina', or 'AzureGovernment'.
    azuread:
      cloud: 'AzurePublic'
      managed_identity:
        client_id: "<client-id of the managed identity>"
      oauth:
        client_id: "<client-id from the Entra app>"
        client_secret: "<client secret from the Entra app>"
        tenant_id: "<Azure subscription tenant Id>"

Prometheus Operator

If you are on a Kubernetes cluster running Prometheus Operator, follow the below steps to send data to your Azure Monitor Workspace.

1. If you are using Microsoft Entra ID authentication, convert the secret using base64 encoding, and apply the secret into your Kubernetes cluster. Save the following into a yaml file. Skip this step if you are using managed identity authentication.

apiVersion: v1
kind: Secret
metadata:
  name: remote-write-secret
  namespace: monitoring # Replace with namespace where Prometheus Operator is deployed. 
type: Opaque
data:
  password: <base64-encoded-secret>

Apply the secret.

# set context to your cluster 
az aks get-credentials -g <aks-rg-name> -n <aks-cluster-name> 

kubectl apply -f <remote-write-secret.yaml>

2. Update the values for remote write section in Prometheus Operator. Copy the following yaml and save it as a file. Refer to Prometheus Operator documentation for more details on the Azure Monitor Workspace remote write specification in Prometheus Operator.

prometheus:
  prometheusSpec:
    remoteWrite:
    - url: "<metrics ingestion endpoint for your Azure Monitor workspace>"
      azureAd:
# AzureAD configuration.
# The Azure Cloud. Options are 'AzurePublic', 'AzureChina', or 'AzureGovernment'.
        cloud: 'AzurePublic'
        managedIdentity:
          clientId: "<clientId of the managed identity>"
        oauth:
          clientId: "<clientId of the Entra app>"
          clientSecret:
            name: remote-write-secret
            key: password
          tenantId: "<Azure subscription tenant Id>"

3. Use helm to update your remote write config using the above yaml file.

helm upgrade -f <YAML-FILENAME>.yml prometheus prometheus-community/kube-prometheus-stack --namespace <namespace where Prometheus Operator is deployed>