how to order sequence of python scrip using batch pool

Renan 20

I have stored three python scripts in Blob Storage and I have created a Batch account and I am about to create the “pool”. But these python scripts has to be run in particular order. Hence my question: how do I order the sequence of the python script to be ran using the batch pool ?

1 answer

Sina Salam 11,991 Reputation points

2024-10-23T13:37:16.1766667+00:00
Hello Renan,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you would like to know how you can order sequence of python script using batch pool.

Job dependencies is the best practices to run your Python scripts in a specific order using Azure Batch.

By following the below steps, you will achieve the purpose:

Ensure you have your Batch pool set up with the necessary compute nodes.

Make sure your Python scripts are uploaded to Azure Blob Storage.

Create a job in Azure Batch that will contain the tasks (your Python scripts).

Define Tasks with Dependencies:

Task 1: Create the first task to run the first Python script.

Task 2: Create the second task to run the second Python script and set it to depend on the completion of Task 1.

Task 3: Create the third task to run the third Python script and set it to depend on the completion of Task 2.

The below is an example of the code to sets up a job with three tasks, each dependent on the completion of the previous one. Make sure to replace placeholders like 'your_batch_account_name', 'your_batch_account_key', 'your_batch_account_url', 'your_pool_id', and 'your_blob_storage_url' with your actual Azure details:

from azure.batch import BatchServiceClient from azure.batch.batch_auth import SharedKeyCredentials from azure.batch.models import * # Initialize Batch client batch_client = BatchServiceClient( credentials=SharedKeyCredentials('your_batch_account_name', 'your_batch_account_key'), batch_url='https://your_batch_account_url' ) # Create job job_id = 'my_job' batch_client.job.add(JobAddParameter(id=job_id, pool_info=PoolInformation(pool_id='your_pool_id'))) # Define tasks tasks = [ TaskAddParameter( id='task1', command_line='python script1.py', resource_files=[ResourceFile(http_url='https://your_blob_storage_url/script1.py', file_path='script1.py')] ), TaskAddParameter( id='task2', command_line='python script2.py', resource_files=[ResourceFile(http_url='https://your_blob_storage_url/script2.py', file_path='script2.py')], depends_on=TaskDependencies(task_ids=['task1']) ), TaskAddParameter( id='task3', command_line='python script3.py', resource_files=[ResourceFile(http_url='https://your_blob_storage_url/script3.py', file_path='script3.py')], depends_on=TaskDependencies(task_ids=['task2']) ) ] # Add tasks to job batch_client.task.add_collection(job_id, tasks)

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.
Please sign in to rate this answer.
Mahesh Goud Juvvadi 1,070 Reputation points Microsoft Vendor

2024-10-24T05:55:23.9733333+00:00

Hi Renan,

Just checking in to see if you have got a chance to see the answer posted by @Sina Salam in resolving the issue.

If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.

We're here to help, so if you have any further questions, don't hesitate to ask.

Thank You.

Renan 20 Reputation points

2024-10-24T07:22:47.0966667+00:00

Hi Sina,

Thank you for the explanation and solution.

Just to make sure I have understood it properly.

I will import this Python script with the dependencies to Blob Storage then create the “job” in Batch Account and use Data Factory to point to this Python scripts with the dependencies orchestrate the pipeline ?

Kind regards

Renan 20 Reputation points

2024-10-24T09:49:38.21+00:00

Hi Sina,

Thank you for the explanation and solution.

Just to make sure I have understood it properly.

I will import this Python script with the dependencies to Blob Storage then create the “job” in Batch Account and use Data Factory to point to this Python scripts with the dependencies orchestrate the pipeline ?

Sina Salam 11,991 Reputation points

2024-10-24T20:13:51.1833333+00:00

Hello Renan,

Thank you for your feedback.

Yes, you have understood it correctly!

First, you will upload your Python scripts (with their dependencies) to Azure Blob Storage. Each script will have a corresponding URL that will be used when creating tasks in the Azure Batch job.

Create a Job in Azure Batch, using the Azure Batch SDK (as shown in the Python code), you will create a job that will contain tasks. Each task represents a Python script to be executed in the batch pool. You can set up dependencies between tasks to ensure they run in the desired sequence.

Then, use Azure Data Factory for Orchestration to trigger your Azure Batch job and manage its execution. You can create a pipeline in ADF that points to your Python scripts and their dependencies in Blob Storage. ADF will manage the execution flow, making sure the tasks run in sequence as per the dependencies you set.

Success.

Mahesh Goud Juvvadi 1,070 Reputation points Microsoft Vendor

2024-10-25T10:08:25.5466667+00:00

Hello Renan,

We noticed your feedback that the above answer was not helpful. Thank you for taking time to share feedback.

Could you please see the latest comment which is provided by @Sina Salam in resolving the issue.

If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.We're here to help, so if you have any further questions, don't hesitate to ask.

Thank You.

Renan 20 Reputation points

2024-10-25T14:05:47.8433333+00:00

Hi,

I have created the batch account and the pool. On the Command Line in Task : do I just write the name of the python file I want to be executed ? For example, I have the python file named "dependencies.py". Do I just add that to the Command Line section ?

Sina Salam 11,991 Reputation points

2024-10-25T18:57:28.4633333+00:00

Hello Renan,Thank you for asking for more clarity.If you’re using Azure Batch SDK for Python to define the task, you would specify this command line as the command_line parameter when creating the task. This is how it will look:

task = batch.models.TaskAddParameter( id="MyTask", command_line="python dependencies.py", resource_files=[batch.models.ResourceFile(http_url="<URL_to_blob_storage>/dependencies.py", file_path="dependencies.py")] )

The name of the file, and the path of the file is what your need.

I have noticed you Downvote that the answer is not helpful, please, do not forget to Upvote and accept as an answer.

Thank you.

Renan 20 Reputation points

2024-10-28T14:31:55.81+00:00

Hi,

I have tried and it is not working. Not sure why?

Sina Salam 11,991 Reputation points

2024-10-28T14:36:15.59+00:00

Hello Renan,

This is a working step that are tested. Could you tell me the error that you've gotten?

Success

Renan 20 Reputation points

2024-10-28T15:08:39.3866667+00:00

Hi Sina,

·       I created the blob storage imported my python scripts

·       Create the batch account

Create the pool

·       on the publisher : Microsoft windows server>

·       Offer: window server

On the command line:

task = batch.models.TaskAddParameter(

    id="MyTask",

    command_line__="python dependencies.py"__,

    resource_files=[batch.models.ResourceFile(http_url="<URL_to_blob_storage>/dependencies.py", file_path="dependencies.py")]

)

On the data factory setting in the command box : python dependencies.py

It was just in progress for a long time, it seems to be stuck. And this code does not take too long to run.

Can you advise if I made any mistake on any of these steps ?

Sina Salam 11,991 Reputation points

2024-10-28T16:01:29.9233333+00:00

Hello Renan,

Thank you for your time and feedback.

I reviewed all your feedback, there are little thing that can make this not to work.

You might be having issues with both the command_line parameter syntax and the configuration of the pool nodes, which could be possible mistakes and this is how to fix them as follow:

In your provided code, you used command_line__ instead of command_line. This will cause an error because command_line__ is not a recognized parameter. Check any incorrect syntax error and update your task definition to use command_line as follows:
task = batch.models.TaskAddParameter( id="MyTask", command_line="python dependencies.py", resource_files=[batch.models.ResourceFile(http_url="<URL_to_blob_storage>/dependencies.py", file_path="dependencies.py")] )

You're using Windows Server image, but by default, it may not have Python installed. Make sure Python is available, either:

Switch to a VM image with Python pre-installed (e.g., a Data Science VM image). OR

Add a start task to install Python at pool creation, like the following:
start_task = batch.models.StartTask( command_line="cmd /c powershell -Command \"Invoke-WebRequest -Uri https://www.python.org/ftp/python/3.9.1/python-3.9.1-amd64.exe -OutFile python-installer.exe; Start-Process python-installer.exe -ArgumentList '/quiet InstallAllUsers=1 PrependPath=1' -NoNewWindow -Wait\"", wait_for_success=True ) pool = batch.models.PoolAddParameter( id="my_pool", virtual_machine_configuration=batch.models.VirtualMachineConfiguration( image_reference=batch.models.ImageReference( publisher="MicrosoftWindowsServer", offer="WindowsServer", sku="2019-datacenter", version="latest" ), node_agent_sku_id="batch.node.windows amd64" ), vm_size="STANDARD_D2_v2", target_dedicated_nodes=1, start_task=start_task )

Alternatively, ADF can trigger the Azure Batch job but does not handle task dependencies. Defining the sequence in the ADF command box alone won’t manage task order. So, you can set up dependencies directly in Azure Batch, not just in ADF. Define tasks in the Azure Batch job (as shown in the previous response) with dependencies to control execution order.

My advice on taken too long on task or getting stuck issues for you is to set up Debugging Stuck Tasks: Because, if the task is "stuck" or takes too long, it’s possible the VM is waiting for resources or installation to complete. The best solution:

Check in your Azure portal under Batch Account > Jobs > [Your Job] > Tasks to view logs or errors for the stuck task.

If the start task does not succeed in installing Python, the main task won’t execute.

Just want to clarify that it was not my initial responses and other comment issue.

If you can take your time to review this solution, you will surely have a good success.

Regards.

Mahesh Goud Juvvadi 1,070 Reputation points Microsoft Vendor

2024-10-29T09:26:18.9666667+00:00

Hi Renan,

Could you please see the latest comment which is provided by @Sina Salam in resolving the issue.

If it was helpful, please click "Upvote and Accept Answer" on his post to let us know.

We're here to help, so if you have any further questions, don't hesitate to ask.

Thank You.

Mahesh Goud Juvvadi 1,070 Reputation points Microsoft Vendor

2024-10-30T08:31:27.75+00:00

Hi Renan,

I wanted to check if you had the opportunity to review the information which was provided by @Sina Salam in previous posted comment.

If you feel that your quires have been resolved, please accept the answer by clicking the "Upvote" and "Accept Answer" on the post.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

how to order sequence of python scrip using batch pool

1 answer

Your answer