Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
320 questions
From within the pool I get the following messages:
Please advise accordingly
Python script not being executed in azure batch using azure data factory
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(44.800000000000004, 85%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELV%3C/text%3E%3C/svg%3E)
Luuk van Vliet
276
Reputation points
Hello,
I have tried reproducing below's tutorial from the microsoft website.
Unfortunately the pipeline status remains "in progress".
https://video2.skills-academy.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory
I use the following pool:
and the start task looks as follows
Adf stays in progress
What could possibly go wrong? Does the basic pool not support this simple scenario?
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma 23,781 Reputation points Microsoft Employee2021-01-13T03:29:04.037+00:00 @Luuk van Vliet Thanks for using Microsoft Q&A !!
Have you checked Node status by going to the batch account pool ?
I believe it is in starting state only and thus your ADF is showing "In progress" only.
Sign in to comment
Accepted answer
-
Luuk van Vliet 276 Reputation points
2021-01-13T07:31:18.917+00:00 -
Saurabh Sharma 23,781 Reputation points Microsoft Employee
2021-01-13T17:18:50.133+00:00 @Luuk van Vliet Can you please go to Tasks under your Batch pool and check the stderr.txt file if you are getting any error logged in that file.
-
Saurabh Sharma 23,781 Reputation points Microsoft Employee
2021-01-14T16:29:57.803+00:00 @Luuk van Vliet Just to confirm if you are using Batch Explorer and selecting the required Dsvm Windows as image.
If not, your node is not starting up and you could get error like below in stderr.txt file -
'pip' is not recognized as an internal or external command,operable program or batch file.As Dsvm images are is a customized VM image and contains Python, using this image will resolve your issue. Please let me know if this is the case.
-
Luuk van Vliet 276 Reputation points
2021-01-14T20:54:37.843+00:00 Thanks a lot, I did not select Dsvm Windows but got it working now!
-
Saurabh Sharma 23,781 Reputation points Microsoft Employee
2021-01-14T22:58:33.807+00:00 @Luuk van Vliet Great to hear. Please share if you have followed any other steps to resolve this issue. This will help the community looking for similar errors.
-
Luuk van Vliet 276 Reputation points
2021-01-15T07:33:12.45+00:00 It is also worth mentioning that choosing the right VM for your scenario is crucial. I started lowest tier VM (Basic series) and it took around 5 minutes to start up and run the pipeline, next I used a higher tier (F-series) VM and it run within 30 seconds.
Also you can RDP to the VM and use command prompt there in order to check for packages being installed correctly, very helpful!
Sign in to comment -