Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
330 questions
What are the most concurrent tasks run and nodes provisioning in a single Azure Batch job?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 52%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMW%3C/text%3E%3C/svg%3E)
Mark Whitney
1
Reputation point
We are having trouble finding published limits for the amount of concurrency that can be handled by Azure Batch. Are there any publicly achieved proof points for either concurrent tasks run (max tasks) or total nodes or cores used in an Azure Batch job?
{count} votes
-
vipullag-MSFT 26,306 Reputation points2021-07-13T04:40:00.733+00:00 As documented here, concurrent tasks per compute node is 4 x number of node cores.
You can also find more details on this in below reference documents.https://video2.skills-academy.com/en-us/azure/batch/batch-parallel-node-tasks
I am also checking with internal team on finding published limits for the amount of concurrency that can be handled by Azure Batch. Will get back on this.
-
Mark Whitney 1 Reputation point
2021-07-19T02:36:23.483+00:00 Thanks, I am more interested in the total max task concurrency shown so far in a single Batch workload (either internal at Azure or by a customer), rather than concurrency per core in a Batch cluster.
-
vipullag-MSFT 26,306 Reputation points
2021-08-23T14:39:29.023+00:00 Apologies in delayed response on this. I checked with internal team on this, Below is what I got from them.
Unfortunately, there’s no simple answer to this question.
For any scheduler (Batch or otherwise), the maximum number of concurrently running Tasks per Pool is highly dependent on the customer’s workload. In general, customers will find that they can run more long running tasks concurrently and fewer short running tasks.
As a real example, we currently have customers running 10s of 1000s of tasks concurrently in a single Pool. But that isn’t guaranteed for your workload, you may find that you are able to run a significantly higher or significantly lower number based on their task profile.
Sign in to comment
Sign in to answer