Will AutoScale formula remove node that is running a task

AzureKnwGt-8579 20

I am new to Azure Batch and I understand that it uses VMSS to create pools in the background. If I define an auto scale formula, can it result in a node being removed while a task is running?

Prrudram-MSFT 24,541 Reputation points

2024-09-10T07:01:11.7633333+00:00

Hello @AzureKnwGt-8579

The way the formula is written by the end-user decides the behavior.

When setting up the formula, there's a read-write variable, named $NodeDeallocationOption, which controls the behavior for the removal of nodes from the pool.

By default, this variable is set to "requeue", having tasks to end immediately and be placed back into the job queue for rescheduling.

For a more detailed explanation, please refer to Autoscale compute nodes in an Azure Batch pool - Azure Batch | Microsoft Learn.
Prrudram-MSFT 24,541 Reputation points

2024-09-10T20:17:24.36+00:00

Hello AzureKnwGt-8579

Since you are using the auto-scaling feature provided by Batch, you can refer to my first response above and find a more detailed explanation of the behavior here.

Regarding your other question about deploying a pool in the network of your choice, also known as BYOVNet (Bring Your Own VNet), Batch does not impose restrictions that the account must be configured as BYOS (Bring Your Own Subscription) or in user subscription mode. That said, BYOVNet can also be used with Batch-managed allocation mode. This is a fairly common scenario for Batch, where the customer brings their own VNet while the compute resources are still provisioned in the subscriptions managed by Batch.

I hope this answers your questions. If I have answered your query, please click "Accept as answer" as a token of appreciation

Accepted answer

Prrudram-MSFT 24,541 Reputation points

2024-09-10T07:02:25.8933333+00:00

Hello @AzureKnwGt-8579

Thank you for reaching out to the Microsoft Q&A platform.

Looking at the title of this query, it appears you are inquiring about the Batch pool behavior after defining the auto-scale rule, a feature provided by the VMSS team (see here), rather than the auto-scale feature provided by the Batch team. Is this the scenario you are referring too?

If so, I assume the end-user has to configure the Batch account in user-subscription mode, allowing them to directly manage the VMSS resource (e.g., defining the auto-scale rule for VMSS) after it is provisioned by Batch. Therefore, the answer would differ from my previous response, which was based on the end-user using the auto-scale feature provided by the Batch team.

Could you please provide the specific use case? This will help us answer your question more accurately.If I have answered your query, please click "Accept as answer" as a token of appreciation
Please sign in to rate this answer.
AzureKnwGt-8579 20 Reputation points

2024-09-10T10:19:13.5066667+00:00

I was actually referring to the scenario for auto-formula provided by the Batch team, but this other scenario was useful information as well.

Also, since I want to use user-subscription mode for my use-case I have additional question about that. From the definition of batch service and user sub. pool allocation mode I would guess that in order to deploy pool in network of my choice I need to use user subscription mode. I don't see that prerequisite in the docs https://video2.skills-academy.com/en-us/azure/batch/batch-virtual-network. When I try to configure VNet for a pool in batch acc with batch service mode on, I create pool successfully. But when I create a node it goes into state "Unusable".

Thanks

AzureKnwGt-8579 20 Reputation points

2024-09-10T10:30:13.59+00:00

[Duplicate comment]
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Q&A Assist

2024-09-09T09:06:05.29+00:00

AI generated content. This question contains an answer created with AI from Q&A Assist. Learn more
Yes, it is possible for an autoscale formula to remove a node while a task is running. When automatic scaling decreases the number of compute nodes in a pool, Batch provides a node deallocation option that you can include in your formulas. You can specify that running tasks are stopped immediately and then requeued for execution on another node, or allowed to finish before the node is removed from the pool. However, setting the node deallocation option as taskcompletion or retaineddata will prevent pool resize operations until all tasks have completed, or all task retention periods have expired, respectively.

References:

Nodes and pools in Azure Batch - Automatic scaling policy

Create a formula to automatically scale compute nodes in a Batch pool - Autoscale formulas
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Will AutoScale formula remove node that is running a task

1 additional answer

Your answer