Autoscale online endpoints in Azure Machine Learning

APPLIES TO: Azure CLI ml extension v2 (current)

If you haven't already set the defaults for the Azure CLI, save your default settings. To avoid passing in the values for your subscription, workspace, and resource group multiple times, run this code:

az account set --subscription <subscription ID>
az configure --defaults workspace=<Azure Machine Learning workspace name> group=<resource group>

1. Set the endpoint and deployment names:

   # set your existing endpoint name
   ENDPOINT_NAME=your-endpoint-name
   DEPLOYMENT_NAME=blue
   

2. Get the Azure Resource Manager ID of the deployment and endpoint:

   # ARM id of the deployment
   DEPLOYMENT_RESOURCE_ID=$(az ml online-deployment show -e $ENDPOINT_NAME -n $DEPLOYMENT_NAME -o tsv --query "id")
   # ARM id of the deployment. todo: change to --query "id"
   ENDPOINT_RESOURCE_ID=$(az ml online-endpoint show -n $ENDPOINT_NAME -o tsv --query "properties.\"azureml.onlineendpointid\"")
   # set a unique name for autoscale settings for this deployment. The below will append a random number to make the name unique.
   AUTOSCALE_SETTINGS_NAME=autoscale-$ENDPOINT_NAME-$DEPLOYMENT_NAME-`echo $RANDOM`
   

3. Create the autoscale profile:

   az monitor autoscale create \
     --name $AUTOSCALE_SETTINGS_NAME \
     --resource $DEPLOYMENT_RESOURCE_ID \
     --min-count 2 --max-count 5 --count 2
   

APPLIES TO: Python SDK azure-ai-ml v2 (current)

1. Import the necessary modules:

   from azure.ai.ml import MLClient
   from azure.identity import DefaultAzureCredential
   from azure.mgmt.monitor import MonitorManagementClient
   from azure.mgmt.monitor.models import AutoscaleProfile, ScaleRule, MetricTrigger, ScaleAction, Recurrence, RecurrentSchedule
   import random 
   import datetime 
   

2. Define variables for the workspace, endpoint, and deployment:

   subscription_id = "<YOUR-SUBSCRIPTION-ID>"
   resource_group = "<YOUR-RESOURCE-GROUP>"
   workspace = "<YOUR-WORKSPACE>"
   
   endpoint_name = "<YOUR-ENDPOINT-NAME>"
   deployment_name = "blue"
   

3. Get Azure Machine Learning and Azure Monitor clients:

   credential = DefaultAzureCredential()
   ml_client = MLClient(
       credential, subscription_id, resource_group, workspace
   )
   
   mon_client = MonitorManagementClient(
       credential, subscription_id
   )
   

4. Get the endpoint and deployment objects:

   deployment = ml_client.online_deployments.get(
       deployment_name, endpoint_name
   )
   
   endpoint = ml_client.online_endpoints.get(
       endpoint_name
   )
   

5. Create an autoscale profile:

   # Set a unique name for autoscale settings for this deployment. The following code appends a random number to create a unique name.
   autoscale_settings_name = f"autoscale-{endpoint_name}-{deployment_name}-{random.randint(0,1000)}"
   
   mon_client.autoscale_settings.create_or_update(
       resource_group, 
       autoscale_settings_name, 
       parameters = {
           "location" : endpoint.location,
           "target_resource_uri" : deployment.id,
           "profiles" : [
               AutoscaleProfile(
                   name="my-scale-settings",
                   capacity={
                       "minimum" : 2, 
                       "maximum" : 5,
                       "default" : 2
                   },
                   rules = []
               )
           ]
       }
   )
   

1. In Azure Machine Learning studio, go to your workspace, and select Endpoints from the left menu.

2. In the list of available endpoints, select the endpoint to configure:

   

3. On the Details tab for the selected endpoint, select Configure auto scaling:

   

4. For the Choose how to scale your resources option, select Custom autoscale to begin the configuration.

5. For the Default scale condition option, configure the following values:

   • Scale mode: Select Scale based on a metric.
   • Instance limits > Minimum: Set the value to 2.
   • Instance limits > Maximum: Set the value to 5.
   • Instance limits > Default: Set the value to 2.

   

Leave the configuration pane open. In the next section, you configure the Rules settings.

APPLIES TO: Azure CLI ml extension v2 (current)

az monitor autoscale rule create \
  --autoscale-name $AUTOSCALE_SETTINGS_NAME \
  --condition "CpuUtilizationPercentage > 70 avg 5m" \
  --scale out 2

The rule is part of the my-scale-settings profile, where autoscale-name matches the name portion of the profile. The value of the rule condition argument indicates the rule triggers when "The average CPU consumption among the VM instances exceeds 70% for 5 minutes." When the condition is satisfied, two more VM instances are allocated.

APPLIES TO: Python SDK azure-ai-ml v2 (current)

1. Create the rule definition:

   rule_scale_out = ScaleRule(
       metric_trigger = MetricTrigger(
           metric_name="CpuUtilizationPercentage",
           metric_resource_uri = deployment.id, 
           time_grain = datetime.timedelta(minutes = 1),
           statistic = "Average",
           operator = "GreaterThan", 
           time_aggregation = "Last",
           time_window = datetime.timedelta(minutes = 5), 
           threshold = 70
       ), 
       scale_action = ScaleAction(
           direction = "Increase", 
           type = "ChangeCount", 
           value = 2, 
           cooldown = datetime.timedelta(hours = 1)
       )
   )
   

   This rule refers to the last 5-minute average of the CPUUtilizationpercentage value from the arguments metric_name, time_window, and time_aggregation. When the value of the metric is greater than the threshold of 70, the deployment allocates two more VM instances.

2. Update the my-scale-settings profile to include this rule:

   mon_client.autoscale_settings.create_or_update(
       resource_group, 
       autoscale_settings_name, 
       parameters = {
           "location" : endpoint.location,
           "target_resource_uri" : deployment.id,
           "profiles" : [
               AutoscaleProfile(
                   name="my-scale-settings",
                   capacity={
                       "minimum" : 2, 
                       "maximum" : 5,
                       "default" : 2
                   },
                   rules = [
                       rule_scale_out
                   ]
               )
           ]
       }
   )
   

The following steps continue with the autoscale configuration.

1. For the Rules option, select the Add a rule link. The Scale rule page opens.

2. On the Scale rule page, configure the following values:

   • Metric name: Select CPU Utilization Percentage.
   • Operator: Set to Greater than.
   • Metric threshold: Set the value to 70.
   • Duration (minutes): Set the value to 5.
   • Time grain statistic: Select Average.
   • Operation: Select Increase count by.
   • Instance count: Set the value to 2.

3. Select Add to create the rule:

   

Leave the configuration pane open. In the next section, you adjust the Rules settings.

APPLIES TO: Azure CLI ml extension v2 (current)

az monitor autoscale rule create \
  --autoscale-name $AUTOSCALE_SETTINGS_NAME \
  --condition "CpuUtilizationPercentage < 25 avg 5m" \
  --scale in 1

APPLIES TO: Python SDK azure-ai-ml v2 (current)

1. Create the rule definition:

   rule_scale_in = ScaleRule(
       metric_trigger = MetricTrigger(
           metric_name="CpuUtilizationPercentage",
           metric_resource_uri = deployment.id, 
           time_grain = datetime.timedelta(minutes = 1),
           statistic = "Average",
           operator = "LessThan", 
           time_aggregation = "Last",
           time_window = datetime.timedelta(minutes = 5), 
           threshold = 30
       ), 
       scale_action = ScaleAction(
           direction = "Increase", 
           type = "ChangeCount", 
           value = 1, 
           cooldown = datetime.timedelta(hours = 1)
       )
   )
   

2. Update the my-scale-settings profile to include this rule:

   mon_client.autoscale_settings.create_or_update(
       resource_group, 
       autoscale_settings_name, 
       parameters = {
           "location" : endpoint.location,
           "target_resource_uri" : deployment.id,
           "profiles" : [
               AutoscaleProfile(
                   name="my-scale-settings",
                   capacity={
                       "minimum" : 2, 
                       "maximum" : 5,
                       "default" : 2
                   },
                   rules = [
                       rule_scale_out, 
                       rule_scale_in
                   ]
               )
           ]
       }
   )
   

The following steps adjust the Rules configuration to support a scale in rule.

1. For the Rules option, select the Add a rule link. The Scale rule page opens.

2. On the Scale rule page, configure the following values:

   • Metric name: Select CPU Utilization Percentage.
   • Operator: Set to Less than.
   • Metric threshold: Set the value to 30.
   • Duration (minutes): Set the value to 5.
   • Time grain statistic: Select Average.
   • Operation: Select Decrease count by.
   • Instance count: Set the value to 1.

3. Select Add to create the rule:

   

   If you configure both scale-out and scale-in rules, your rules look similar to the following screenshot. The rules specify that if average CPU load exceeds 70% for 5 minutes, two more nodes should be allocated, up to the limit of five. If CPU load is less than 30% for 5 minutes, a single node should be released, down to the minimum of two.

   

Leave the configuration pane open. In the next section, you specify other scale settings.

APPLIES TO: Azure CLI ml extension v2 (current)

az monitor autoscale rule create \
 --autoscale-name $AUTOSCALE_SETTINGS_NAME \
 --condition "RequestLatency > 70 avg 5m" \
 --scale out 1 \
 --resource $ENDPOINT_RESOURCE_ID

APPLIES TO: Python SDK azure-ai-ml v2 (current)

1. Create the rule definition:

   rule_scale_out_endpoint = ScaleRule(
       metric_trigger = MetricTrigger(
           metric_name="RequestLatency",
           metric_resource_uri = endpoint.id, 
           time_grain = datetime.timedelta(minutes = 1),
           statistic = "Average",
           operator = "GreaterThan", 
           time_aggregation = "Last",
           time_window = datetime.timedelta(minutes = 5), 
           threshold = 70
       ), 
       scale_action = ScaleAction(
           direction = "Increase", 
           type = "ChangeCount", 
           value = 1, 
           cooldown = datetime.timedelta(hours = 1)
       )
   )
   

   This rule's metric_resource_uri field now refers to the endpoint rather than the deployment.

2. Update the my-scale-settings profile to include this rule:

   mon_client.autoscale_settings.create_or_update(
       resource_group, 
       autoscale_settings_name, 
       parameters = {
           "location" : endpoint.location,
           "target_resource_uri" : deployment.id,
           "profiles" : [
               AutoscaleProfile(
                   name="my-scale-settings",
                   capacity={
                       "minimum" : 2, 
                       "maximum" : 5,
                       "default" : 2
                   },
                   rules = [
                       rule_scale_out, 
                       rule_scale_in,
                       rule_scale_out_endpoint
                   ]
               )
           ]
       }
   )
   

The following steps continue the rule configuration on the Custom autoscale page.

1. At the bottom of the page, select the Add a scale condition link.

2. On the Scale condition page, select Scale based on metric, and then select the Add a rule link. The Scale rule page opens.

3. On the Scale rule page, configure the following values:

   • Metric source: Select Other resource.
   • Resource type: Select Machine Learning online endpoints.
   • Resource: Select your endpoint.
   • Metric name: Select Request latency.
   • Operator: Set to Greater than.
   • Metric threshold: Set the value to 70.
   • Duration (minutes): Set the value to 5.
   • Time grain statistic: Select Average.
   • Operation: Select Increase count by.
   • Instance count: Set the value to 1.

4. Select Add to create the rule:

   

APPLIES TO: Azure CLI ml extension v2 (current)

az monitor autoscale profile create \
  --name weekend-profile \
  --autoscale-name $AUTOSCALE_SETTINGS_NAME \
  --min-count 2 --count 2 --max-count 2 \
  --recurrence week sat sun --timezone "Pacific Standard Time"

APPLIES TO: Python SDK azure-ai-ml v2 (current)

mon_client.autoscale_settings.create_or_update(
    resource_group, 
    autoscale_settings_name, 
    parameters = {
        "location" : endpoint.location,
        "target_resource_uri" : deployment.id,
        "profiles" : [
            AutoscaleProfile(
                name="Default",
                capacity={
                    "minimum" : 2, 
                    "maximum" : 2,
                    "default" : 2
                },
                recurrence = Recurrence(
                    frequency = "Week", 
                    schedule = RecurrentSchedule(
                        time_zone = "Pacific Standard Time", 
                        days = ["Saturday", "Sunday"], 
                        hours = [], 
                        minutes = []
                    )
                )
            )
        ]
    }
)

The following steps configure the rule with options on the Custom autoscale page in the studio.

1. At the bottom of the page, select the Add a scale condition link.

2. On the Scale condition page, select Scale to a specific instance count, and then select the Add a rule link. The Scale rule page opens.

3. On the Scale rule page, configure the following values:

   • Instance count: Set the value to 2.
   • Schedule: Select Repeat specific days.
   • Set the schedule pattern: Select Repeat every and Saturday and Sunday.

4. Select Add to create the rule:

   

APPLIES TO: Azure CLI ml extension v2 (current)

az monitor autoscale update \
  --autoscale-name $AUTOSCALE_SETTINGS_NAME \
  --enabled false

APPLIES TO: Python SDK azure-ai-ml v2 (current)

mon_client.autoscale_settings.create_or_update(
    resource_group, 
    autoscale_settings_name, 
    parameters = {
        "location" : endpoint.location,
        "target_resource_uri" : deployment.id,
        "enabled" : False
    }
)

• To disable an autoscale profile in use, select Manual scale, and then select Save.

• To enable an autoscale profile, select Custom autoscale. The studio lists all recognized autoscale profiles for the workspace. Select a profile and then select Save to enable.

APPLIES TO: Azure CLI ml extension v2 (current)

# delete the autoscaling profile
az monitor autoscale delete -n "$AUTOSCALE_SETTINGS_NAME"

# delete the endpoint
az ml online-endpoint delete --name $ENDPOINT_NAME --yes --no-wait

APPLIES TO: Python SDK azure-ai-ml v2 (current)

mon_client.autoscale_settings.delete(
    resource_group, 
    autoscale_settings_name
)

ml_client.online_endpoints.begin_delete(endpoint_name)

1. In Azure Machine Learning studio, go to your workspace and select Endpoints from the left menu.

2. In the list of endpoints, select the endpoint to delete (check the circle next to the model name).

3. Select Delete.

Alternatively, you can delete a managed online endpoint directly in the endpoint details page.