Replicating a Pipeline Multiple Times in Azure Synapse Analytics

Aditya Singh 135 Reputation points
2024-08-16T09:26:31.9833333+00:00

How can I replicate a pipeline multiple times in Azure Synapse Analytics? I need to replicate my pipeline 10 times with only a change to the table name which is passed as a parameter.

Thanks in advance!

Azure SQL Database
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,463 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
3,105 questions
Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,859 questions
Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,162 questions
{count} votes

1 answer

Sort by: Most helpful
  1. NIKHILA NETHIKUNTA 1,760 Reputation points Microsoft Vendor
    2024-08-19T09:16:13.3866667+00:00

    Hi @Aditya Singh
    Thank you for your question and using Microsoft Q&A platform.

    To replicate your pipeline multiple times in Azure Synapse Analytics with a different table name passed as a parameter for each instance, you can follow these steps:

    1. In your existing pipeline, create a parameter for the table name. Use this parameter in your activities. For example, I have a pipeline named 'ifelsepipeline' and I have created a parameter named 'tableName'.

    User's image

    I have used this parameter inside the lookup activity. User's image

    1. Create a new pipeline that will act as a master pipeline. Add a ForEach activity to the master pipeline. This activity will iterate over a list of table names. Here I have created a new pipeline named 'ExecutePipeline'.

    Add a variable of type 'Array' which contains the list of table names.

    User's image

    1. Inside the ForEach activity, use an Execute Pipeline activity to call your original pipeline. Pass the table name parameter to the pipeline during each iteration of the ForEach loop.

    User's image

    1. Execute the master pipeline. It will run the original pipeline 3 times in my case as I have given 3 table names, each time with a different table name. User's image For more information related to Foreach activity please refer to this document:
      https://video2.skills-academy.com/en-us/azure/data-factory/control-flow-for-each-activity

    Hope this helps. Do let us know if you any further queries.


    If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.