Metrics export through data collection rules (Preview)

Data collection rules (DCRs) are used to collect monitoring data from your Azure resources. For a list of data collection scenario, see Data collection rules - Overview. You can now use DCRs to collect and export platform metrics.

Currently, platform metrics can be collected using both DCR and Diagnostic Settings. A growing number of resources support metrics export using DCRs. See Supported resources and regions for a list of supporting resources.

Using DCRs to export metrics provides the following advantages over diagnostic settings:

• DCR configuration enables exporting metrics with dimensions.
• DCR configuration enables filtering based on metric name - so that you can export only the metrics that you need.
• DCRs are more flexible and scalable compared to Diagnostic Settings.
• End to end latency for DCRs is within 3 minutes. This is a major improvement over Diagnostic Settings where metrics export latency is 6-10 minutes.

Use metrics export through DCRs for continuous export of metrics data. For querying historical data, use the Data plane Metrics Batch API. For a comparison of the two services, see Data plane Metrics Batch API query versus Metrics export.

Create DCRs for metrics using the REST API, Azure CLI, or Azure PowerShell. For information on how to create DCRs for metrics export, see Create data collection rules for metrics.

When you create a DCR, you must create a Data collection rule association (DCRA) to associate the DCR with the resource to be monitored. You can create a single DCR for many resource types. For information on how to create a DCRA see Create data collection rule associations. When using the Azure portal, the DCRA is created automatically.

Export destinations

Metrics can be exported to one of the following destinations per DCR:

• Log Analytics workspaces.
  Exporting to Log Analytics workspaces can be across regions. The Log Analytics workspace and the DCR must be in the same region but resources that are being monitored can be in any region. Metrics sent to a log analytics workspace are stored in the AzureMetricsV2 table.

• Azure storage accounts.
  The storage account, the DCR, and the resources being monitored must all be in the same region.

• Event Hubs.
  The Event Hubs, the DCR, and the resources being monitored must all be in the same region.

For a sample of the data in each destination, see Exported data.

Limitations

DCRs for metrics export have the following limitations:

• Only one destination type can be specified per DCR.
• A maximum of 5 DCRs can be associated with a single Azure Resource.

Supported resources and regions

The following resources support metrics export using data collection rules:

Supported regions

You can create a DCR for metrics export in any region, but the resources that you want to export metrics from must be in one of the following regions:

• Australia East
• Central US
• CentralUsEuap
• South Central US
• East US
• East US 2
• Eastus2Euap
• West US
• West US 2
• North Europe
• West Europe
• UK South

DCR specifications

Data collection rules are defined in a JSON object. The following properties are required to create a DCR for metrics export.

Only one destination type can be specified per DCR. Remove the unwanted destinations when copying the following example.

{
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                    // a list of resource types and metrics to collect metrics from
                        "<resourcetype>:<metric name> | Metrics-Group-All", 
                        "<resourcetype>:<metric name> | Metrics-Group-All"
                    ],
                    "name": "<data sources name>"
                }
            ]
        },
        "destinations": {
            // Choose a single destination type of either logAnalytics, storageAccounts, or eventHubs
            "logAnalytics": [
                {
                    "workspaceResourceId": "workspace Id",
                    "name": "<destination name>"
                }
            ],    
            "storageAccounts": [
                {
                    "storageAccountResourceId": "<storage account Id>", 
                    "containerName": "<container name>",
                    "name": "<destination name>"
                }
            ],
            "eventHubs": [ 
                 { 
                    "eventHubResourceId": "event hub id", 
                     "name": "<destination name>" 
                 } 
             ],     },
        "dataFlows": [
            {
                "streams": [
                 // a list of resource types and metrics to pass to the destination
                        "<resourcetype>:<metric name> | Metrics-Group-All", 
                        "<resourcetype>:<metric name> | Metrics-Group-All"               
                         ],
                "destinations": [
                    "<destination name>"               
                ]
            }
        ]
    },
    // identity is required for Storage Account and Event Hubs destinations
    "identity": {
         "type": "userassigned", 
         "userAssignedIdentities": { 
					 "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/rg-001/providers/Microsoft.ManagedIdentity/userAssignedIdentities/DRCIdentity": {} 
         }    
    },
"kind": "PlatformTelemetry",
    "location": "eastus"
}

User and system assigned identities

Both user and system assigned identities are supported when creating DCRs. An identity is required for Storage Account and Event Hubs destinations. You can use a system assigned or user assigned identity. For more information, see Assign Azure roles to a managed identity.

To use a system assigned identity, add the identity object as follows:

    "identity": {
         "type": "systemAssigned"
    },

To use a user assigned identity, add the identity object as follows:

    "identity": {
         "type": "userassigned", 

         "userAssignedIdentities": { 
           "/subscriptions/<subscriptionId>/resourceGroups/<resource group name>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<identity name>": {} 
         }    

    }

Filtering metrics

When specifying the metrics to export, you can filter the metrics by name or request all metrics by using Metrics-Group-All. For a list of supported metrics, see Supported metrics and log categories by resource type.

The following example shows how to filter metrics by name.

{
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                        "Microsoft.Compute/virtualMachines:Percentage CPU",
                        "Microsoft.Compute/virtualMachineScaleSets:Percentage CPU",
                        "Microsoft.Cache/redis:Cache Hits"
                    ],
                    "name": "myPlatformTelemetryDataSource"
                }
            ]
        },
        "destinations": {
            "logAnalytics": [
                {
                    "workspaceResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/arg-001/providers/microsoft.operationalinsights/workspaces/loganalyticsworkspace001",
                    "name": "destinationName"
                }
            ]
        },
        "dataFlows": [
            {
                "streams": [
                    "Microsoft.Compute/virtualMachines:Percentage CPU",
                    "Microsoft.Compute/virtualMachineScaleSets:Percentage CPU",
                    "Microsoft.Cache/redis:Cache Hits"
                ],
                "destinations": [
                    "destinationName"
                ]
            }
        ]
    },
    "kind": "PlatformTelemetry",
    "location": "eastus"
}

Sample metrics export JSON objects

The following examples show sample DCR JSON objects for metrics export to each destination type.

Log Analytics workspaces

The following example shows a data collection rule for metrics that sends specific metrics from virtual machines, virtual machine scale sets, and all key vault metrics to a Log Analytics workspace:

{
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                        "Microsoft.Compute/virtualMachines:Percentage CPU",
                        "Microsoft.Compute/virtualMachineScaleSets:Available Memory Bytes",
                         "Microsoft.KeyVault/vaults:Metrics-Group-All"
                    ],
                    "name": "myPlatformTelemetryDataSource"
                }
            ]
        },
        "destinations": {
            "logAnalytics": [ 
                { 
                    "workspaceResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-001/providers/microsoft.operationalinsights/workspaces/laworkspace001", 
                    "name": "ladestination" 
                } 
            ] 
        },
        "dataFlows": [
            {
                "streams": [
                        "Microsoft.Compute/virtualMachines:Percentage CPU",
                        "Microsoft.Compute/virtualMachineScaleSets:Available Memory Bytes",
                        "Microsoft.KeyVault/vaults:Metrics-Group-All"
                        ],    
                "destinations": [
                    "ladestination"
                    
                ]
            }
        ]
    },

"kind": "PlatformTelemetry",
    "location": "centralus"
}

Storage accounts

The following example shows a data collection rule for metrics that sends all metrics from virtual machines, virtual machine scale sets, Redis cache, and Key Vaults to a storage account.

{
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                        "Microsoft.Compute/virtualMachines:Metrics-Group-All",
                        "Microsoft.Compute/virtualMachineScaleSets:Metrics-Group-All",
                        "Microsoft.Cache/redis:Metrics-Group-All",
                        "Microsoft.keyvault/vaults:Metrics-Group-All"
                    ],
                    "name": "myPlatformTelemetryDataSource"
                }
            ]
        },
        "destinations": {
            "storageAccounts": [
                {
                    "storageAccountResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-001/providers/Microsoft.Storage/storageAccounts/metricsexport001",
                    "containerName": "metritcs-001",
                    "name": "desitnationName"
                }
            ]
        },
        "dataFlows": [
            {
                "streams": [
                    "Microsoft.Compute/virtualMachines:Metrics-Group-All",
                    "Microsoft.Compute/virtualMachineScaleSets:Metrics-Group-All",
                    "Microsoft.Cache/redis:Metrics-Group-All",
                    "Microsoft.keyvault/vaults:Metrics-Group-All"
                ],
                "destinations": [
                    "desitnationName"
                ]
            }
        ]
    },
    "identity": {
         "type": "systemAssigned"
    },
"kind": "PlatformTelemetry",
    "location": "eastus2"
}

Event Hubs

The following example shows a data collection rule for metrics export that sends all metrics from virtual machines, and the ServiceApiHit metric from Key Vaults to Event Hubs.

{
    "properties": {
        "dataSources": {
            "platformTelemetry": [
                {
                    "streams": [
                        "Microsoft.Compute/virtualMachines:Metrics-Group-All",
                        "Microsoft.keyvault/vaults:ServiceApiHit"
                    ],
                    "name": "myPlatformTelemetryDataSource"
                }
            ]
        },
        "destinations": {
            "eventHubs": [ 
                { 
                    "eventHubResourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourceGroups/rg-001/providers/Microsoft.EventHub/namespaces/event-hub-001/eventhubs/hub-001", 
                    "name": "hub1" 
                } 
           ] 
        },
        "dataFlows": [
            {
                "streams": [
                    "Microsoft.Compute/virtualMachines:Metrics-Group-All",
                    "Microsoft.keyvault/vaults:ServiceApiHit"
                ],
                "destinations": [
                    "hub1"
                ]
            }
        ]
    },
    "identity": {
         "type": "systemAssigned"
    },
"kind": "PlatformTelemetry",
    "location": "eastus"
}

Create DCRs for metrics export

Create DCRs for metrics export using the Azure portal, CLI, PowerShell, REST API, or ARM template. For more information, see Create and edit data collection rules (DCRs) and associations in Azure Monitor.

Exported data

The following examples show the data exported to each destination.

Log analytics workspaces

Data exported to a Log Analytics workspace is stored in the AzureMetricsV2 table in the Log Analytics workspace in the following format:

For example:

Storage accounts

The following example shows data exported to a storage account:

{
    "Average": "31.5",
    "Count": "2",
    "Maximum": "52",
    "Minimum": "11",
    "Total": "63",
    "resourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-dcrs/providers/microsoft.keyvault/vaults/dcr-vault",
    "time": "2024-08-20T14:13:00.0000000Z",
    "unit": "MilliSeconds",
    "metricName": "ServiceApiLatency",
    "timeGrain": "PT1M",
    "dimension": {
        "ActivityName": "vaultget",
        "ActivityType": "vault",
        "StatusCode": "200",
        "StatusCodeClass": "2xx"
    }
}

Event Hubs

The following example shows a metric exported to Event Hubs.

    {
      "Average": "1",
      "Count": "1",
      "Maximum": "1",
      "Minimum": "1",
      "Total": "1",
      "resourceId": "/subscriptions/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/resourcegroups/rg-dcrs/providers/microsoft.keyvault/vaults/dcr-vault",
      "time": "2024-08-22T13:43:00.0000000Z",
      "unit": "Count",
      "metricName": "ServiceApiHit",
      "timeGrain": "PT1M",
      "dimension": {
        "ActivityName": "keycreate",
        "ActivityType": "key"
      },
      "EventProcessedUtcTime": "2024-08-22T13:49:17.1233030Z",
      "PartitionId": 0,
      "EventEnqueuedUtcTime": "2024-08-22T13:46:04.5570000Z"
    }

Next steps

• Create and edit data collection rules
• Data plane metrics batch API query versus Metrics Export
• Data collection rules, overview
• Best practices for data collection rule creation and management in Azure Monitor