Using Application Health extension with Virtual Machine Scale Sets

Monitoring your application health is an important signal for managing and upgrading your deployment. Azure Virtual Machine Scale Sets provide support for Rolling Upgrades including Automatic OS-Image Upgrades and Automatic VM Guest Patching, which rely on health monitoring of the individual instances to upgrade your deployment. You can also use Application Health Extension to monitor the application health of each instance in your scale set and perform instance repairs using Automatic Instance Repairs.

This article describes how you can use the two types of Application Health extension, Binary Health States or Rich Health States, to monitor the health of your applications deployed on Virtual Machine Scale Sets.

Prerequisites

This article assumes that you're familiar with:

• Azure virtual machine extensions
• Modifying Virtual Machine Scale Sets

When to use the Application Health extension

The Application Health Extension is deployed inside a Virtual Machine Scale Set instance and reports on application health from inside the scale set instance. The extension probes on a local application endpoint and will update the health status based on TCP/HTTP(S) responses received from the application. This health status is used by Azure to initiate repairs on unhealthy instances and to determine if an instance is eligible for upgrade operations.

The extension reports health from within a VM and can be used in situations where an external probe such as the Azure Load Balancer health probes can’t be used.

Binary versus Rich Health States

Application Health Extensions has two options available: Binary Health States and Rich Health States. The following table highlights some key differences between the two options. See the end of this section for general recommendations.

¹ The Unknown state is unavailable on TCP protocol. ² Only applicable for HTTP/HTTPS protocol. TCP protocol will follow the same process of identifying Unhealthy instances as in Binary Health States.

In general, you should use Binary Health States if:

• You're not interested in configuring custom logic to identify and flag an unhealthy instance
• You don't require an initializing grace period for newly created instances

You should use Rich Health States if:

• You send health signals through HTTP/HTTPS protocol and can submit health information through the probe response body
• You would like to use custom logic to identify and mark unhealthy instances
• You would like to set an initializing grace period for newly created instances, so that they settle into a steady Health State before making the instance eligible for rolling upgrade or instance repairs

Binary Health States

Binary Health State reporting contains two Health States, Healthy and Unhealthy. The following tables provide a brief description for how the Health States are configured.

HTTP/HTTPS Protocol

TCP Protocol

Some scenarios that may result in an Unhealthy state include:

• When the application endpoint returns a non-200 status code
• When there's no application endpoint configured inside the virtual machine instances to provide application health status
• When the application endpoint is incorrectly configured
• When the application endpoint isn't reachable

Rich Health States

Rich Health States reporting contains four Health States, Initializing, Healthy, Unhealthy, and Unknown. The following tables provide a brief description for how each Health State is configured.

HTTP/HTTPS Protocol

TCP Protocol

Initializing state

This state only applies to Rich Health States. The Initializing state only occurs once at extension start time and can be configured by the extension settings gracePeriod and numberOfProbes.

At extension startup, the application health will remain in the Initializing state until one of two scenarios occurs:

• The same Health State (Healthy or Unhealthy) is reported a consecutive number of times as configured through numberOfProbes
• The gracePeriod expires

If the same Health State (Healthy or Unhealthy) is reported consecutively, the application health will transition out of the Initializing state and into the reported Health State (Healthy or Unhealthy).

Example

If numberOfProbes = 3, that would mean:

• To transition from Initializing to Healthy state: Application health extension must receive three consecutive Healthy signals via HTTP/HTTPS or TCP protocol
• To transition from Initializing to Unhealthy state: Application health extension must receive three consecutive Unhealthy signals via HTTP/HTTPS or TCP protocol

If the gracePeriod expires before a consecutive health status is reported by the application, the instance health will be determined as follows:

• HTTP/HTTPS protocol: The application health will transition from Initializing to Unknown
• TCP protocol: The application health will transition from Initializing to Unhealthy

Unknown state

This state only applies to Rich Health States. The Unknown state is only reported for "http" or "https" probes and occurs in the following scenarios:

• When a non-2xx status code is returned by the application
• When the probe request times out
• When the application endpoint is unreachable or incorrectly configured
• When a missing or invalid value is provided for ApplicationHealthState in the response body
• When the grace period expires

An instance in an Unknown state is treated similar to an Unhealthy instance. If enabled, instance repairs will be carried out on an Unknown instance while rolling upgrades will be paused until the instance falls back into a Healthy state.

The following table shows the health status interpretation for Rolling Upgrades and Instance Repairs:

Extension schema for Binary Health States

The following JSON shows the schema for the Application Health extension. The extension requires at a minimum either a "tcp", "http" or "https" request with an associated port or request path respectively.

{
  "extensionProfile" : {
     "extensions" : [
      {
        "name": "HealthExtension",
        "properties": {
          "publisher": "Microsoft.ManagedServices",
          "type": "<ApplicationHealthLinux or ApplicationHealthWindows>",
          "autoUpgradeMinorVersion": true,
          "typeHandlerVersion": "1.0",
          "settings": {
            "protocol": "<protocol>",
            "port": <port>,
            "requestPath": "</requestPath>",
            "intervalInSeconds": 5,
            "numberOfProbes": 1
          }
        }
      }
    ]
  }
} 

Property values

Settings

Extension schema for Rich Health States

The following JSON shows the schema for the Rich Health States extension. The extension requires at a minimum either an "http" or "https" request with an associated port or request path respectively. TCP probes are also supported, but won't be able to set the ApplicationHealthState through the probe response body and won't have access to the Unknown state.

{
  "extensionProfile" : {
     "extensions" : [
      {
        "name": "HealthExtension",
        "properties": {
          "publisher": "Microsoft.ManagedServices",
          "type": "<ApplicationHealthLinux or ApplicationHealthWindows>",
          "autoUpgradeMinorVersion": true,
          "typeHandlerVersion": "2.0",
          "settings": {
            "protocol": "<protocol>",
            "port": <port>,
            "requestPath": "</requestPath>",
            "intervalInSeconds": 5,
            "numberOfProbes": 1,
            "gracePeriod": 600
          }
        }
      }
    ]
  }
} 

Property values

Settings

Deploy the Application Health extension

There are multiple ways of deploying the Application Health extension to your scale sets as detailed in the following examples.

Binary Health States

PUT on `/subscriptions/subscription_id/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachineScaleSets/myScaleSet/extensions/myHealthExtension?api-version=2018-10-01`

{
  "name": "myHealthExtension",
  "location": "<location>", 
  "properties": {
    "publisher": "Microsoft.ManagedServices",
    "type": "ApplicationHealthWindows",
    "autoUpgradeMinorVersion": true,
    "typeHandlerVersion": "1.0",
    "settings": {
      "protocol": "<protocol>",
      "port": <port>,
      "requestPath": "</requestPath>"
    }
  }
}

POST on `/subscriptions/<subscriptionId>/resourceGroups/<myResourceGroup>/providers/Microsoft.Compute/virtualMachineScaleSets/< myScaleSet >/manualupgrade?api-version=2022-08-01`

{
  "instanceIds": ["*"]
}

# Define the scale set variables
$vmScaleSetName = "myVMScaleSet"
$vmScaleSetResourceGroup = "myVMScaleSetResourceGroup"

# Define the Application Health extension properties
$publicConfig = @{"protocol" = "http"; "port" = 80; "requestPath" = "/healthEndpoint"};
$extensionName = "myHealthExtension"
$extensionType = "ApplicationHealthWindows"
$publisher = "Microsoft.ManagedServices"

# Get the scale set object
$vmScaleSet = Get-AzVmss `
  -ResourceGroupName $vmScaleSetResourceGroup `
  -VMScaleSetName $vmScaleSetName

# Add the Application Health extension to the scale set model
Add-AzVmssExtension -VirtualMachineScaleSet $vmScaleSet `
  -Name $extensionName `
  -Publisher $publisher `
  -Setting $publicConfig `
  -Type $extensionType `
  -TypeHandlerVersion "1.0" `
  -AutoUpgradeMinorVersion $True

# Update the scale set
Update-AzVmss -ResourceGroupName $vmScaleSetResourceGroup `
  -Name $vmScaleSetName `
  -VirtualMachineScaleSet $vmScaleSet
  
# Upgrade instances to install the extension
Update-AzVmssInstance -ResourceGroupName $vmScaleSetResourceGroup `
  -VMScaleSetName $vmScaleSetName `
  -InstanceId '*'

az vmss extension set \
  --name ApplicationHealthLinux \
  --publisher Microsoft.ManagedServices \
  --version 1.0 \
  --resource-group <myVMScaleSetResourceGroup> \
  --vmss-name <myVMScaleSet> \
  --settings ./extension.json

{
  "protocol": "<protocol>",
  "port": <port>,
  "requestPath": "</requestPath>"
}

az vmss update-instances \
  --resource-group <myVMScaleSetResourceGroup> \
  --name <myVMScaleSet> \
  --instance-ids "*"

Rich Health States

PUT on `/subscriptions/subscription_id/resourceGroups/myResourceGroup/providers/Microsoft.Compute/virtualMachineScaleSets/myScaleSet/extensions/myHealthExtension?api-version=2018-10-01`

{
  "name": "myHealthExtension",
  "location": "<location>",
  "properties": {
    "publisher": "Microsoft.ManagedServices",
    "type": "ApplicationHealthWindows",
    "autoUpgradeMinorVersion": true,
    "typeHandlerVersion": "2.0",
    "settings": {
      "protocol": "<protocol>",
      "port": <port>,
      "requestPath": "</requestPath>",
      "intervalInSeconds": <intervalInSeconds>,
      "numberOfProbes": <numberOfProbes>,
      "gracePeriod": <gracePeriod>
    }
  }
}

POST on `/subscriptions/<subscriptionId>/resourceGroups/<myResourceGroup>/providers/Microsoft.Compute/virtualMachineScaleSets/< myScaleSet >/manualupgrade?api-version=2022-08-01`

{
  "instanceIds": ["*"]
}

# Define the scale set variables
$vmScaleSetName = "myVMScaleSet"
$vmScaleSetResourceGroup = "myVMScaleSetResourceGroup"

# Define the Application Health extension properties
$publicConfig = @{"protocol" = "http"; "port" = 80; "requestPath" = "/healthEndpoint"; "gracePeriod" = 600};
$extensionName = "myHealthExtension"
$extensionType = "ApplicationHealthWindows"
$publisher = "Microsoft.ManagedServices"

# Get the scale set object
$vmScaleSet = Get-AzVmss `
  -ResourceGroupName $vmScaleSetResourceGroup `
  -VMScaleSetName $vmScaleSetName

# Add the Application Health extension to the scale set model
Add-AzVmssExtension -VirtualMachineScaleSet $vmScaleSet `
  -Name $extensionName `
  -Publisher $publisher `
  -Setting $publicConfig `
  -Type $extensionType `
  -TypeHandlerVersion "2.0" `
  -AutoUpgradeMinorVersion $True

# Update the scale set
Update-AzVmss -ResourceGroupName $vmScaleSetResourceGroup `
  -Name $vmScaleSetName `
  -VirtualMachineScaleSet $vmScaleSet
  
# Upgrade instances to install the extension
Update-AzVmssInstance -ResourceGroupName $vmScaleSetResourceGroup `
  -VMScaleSetName $vmScaleSetName `
  -InstanceId '*'

az vmss extension set \
  --name ApplicationHealthLinux \
  --publisher Microsoft.ManagedServices \
  --version 2.0 \
  --resource-group <myVMScaleSetResourceGroup> \
  --vmss-name <myVMScaleSet> \
  --settings ./extension.json

{
  "protocol": "<protocol>",
  "port": <port>,
  "requestPath": "</requestPath>",
  "gracePeriod": <healthExtensionGracePeriod>
}

az vmss update-instances \
  --resource-group <myVMScaleSetResourceGroup> \
  --name <myVMScaleSet> \
  --instance-ids "*"

Troubleshoot

Need help configuring a probe response

See application health samples for examples of health probe responses being emitted to a local endpoint.

View VMHealth - single instance

Get-AzVmssVM 
  -InstanceView `
  -ResourceGroupName <rgName> `
  -VMScaleSetName <vmssName> `
  -InstanceId <instanceId> 

View VMHealth – batch call

This is only available for Virtual Machine Scale Sets with Uniform orchestration.

GET on `/subscriptions/<subscriptionID>/resourceGroups/<resourceGroupName>/providers/Microsoft.Compute/virtualMachineScaleSets/<vmssName>/virtualMachines/?api-version=2022-03-01&$expand=instanceview`

Health State isn't showing up

If Health State isn't showing up in Azure portal or via GET call, check to ensure that the VM is upgraded to the latest model. If the VM isn't on the latest model, upgrade the VM and the health status will come up.

Extension execution output log

Extension execution output is logged to files found in the following directories:

C:\WindowsAzure\Logs\Plugins\Microsoft.ManagedServices.ApplicationHealthWindows\<version>\

/var/lib/waagent/Microsoft.ManagedServices.ApplicationHealthLinux-<extension_version>/status
/var/log/azure/applicationhealth-extension

The logs also periodically capture the application health status.

Next steps

Learn how to deploy your application on Virtual Machine Scale Sets.