Running specific pipelines from an Orchestration pipeline

Hello ADITYA SINGH,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you would like to choose the pipeline of your choice to run separate from the others, while individual pipelines from an Orchestration pipeline that has multiple pipelines running in parallel.

Solution

In Azure Data Factory, you can orchestrate multiple pipelines using a pipeline activity called Execute Pipeline. Therefore, if you want to run individual pipelines separately from an orchestration pipeline, you can do so by manually or automatically triggering them from the Azure Data Factory interface or by using Azure Data Factory REST API, PowerShell, or Azure CLI.

You can manually trigger a pipeline from the Azure Data Factory interface:

1. Navigate to the Author tab on the left-hand menu.
2. In the factory resources box, click on the Pipelines section.
3. Click on the pipeline you want to run.
4. Click on the Add Trigger button at the top and select Trigger Now.

If you want to automate the triggering process, you can use Azure Data Factory REST API, PowerShell, or Azure CLI. The below is an example of how to trigger a pipeline using Azure CLI:

az datafactory pipeline create-run --resource-group <resource-group-name> --factory-name <data-factory-name> --pipeline-name <pipeline-name>

Remember to replace <resource-group-name>, <data-factory-name>, and <pipeline-name> with your actual resource group name, data factory name, and pipeline name respectively.

NOTE:

Please note that running individual pipelines separately from an orchestration pipeline might not maintain the dependencies between the pipelines. If there are dependencies, you might need to manage them manually.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam