How long is it supposed to take an Azure batch node to start running a task?

Eric 146 Reputation points
2021-04-20T05:42:10.5+00:00

I'm trying to run an R script in paralell on azure batch. The script uses the xgboost package and pulls csv files from my azure storage account. The output is a csv file back in to my azure storage account. After a node boots it literally takes hours for my tasks to go from queued to running. This happens whether I use dedicated nodes or low priority nodes. If I add .packages = c('xgboost') to my foreach function Rstudio tells me "Job Preparation Status: Package(s) being installed" and that step takes literally hours to finish even if I only have 1 node in the pool.

I understand cloud computing resources take time to boot up, but is it supposed to take hours or is there something I can do to speed it up? It takes about 20 minutes for my PC to run 1 task on it's own.

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
320 questions
Accepted answer
  1. prmanhas-MSFT 17,901 Reputation points Microsoft Employee
    2021-04-23T05:43:56.993+00:00

    @Eric Thank you for your patience over the matter!!!

    Below is the response I got from our internal team:

    It can take 2 – 6 mins for VMs to be started and in a position to run Batch tasks. It isn’t supposed to take hours.

    You might be using doAzureParallel which might be the reason as well. Whether they are or aren’t, the customer should submit a support ticket so someone can help them.

    I would recommend you to open Support Request with Microsoft Support Team if you have a support plan in place if not do let me know I can help you open a One Time Free Technical Support.

    Hope it helps!!!

    Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest