How can I add pagination as a parameter for the Azure Data Factory Rest API call to retrieve all data?

To add pagination parameters to the Azure Data Factory REST API calls, you'll need to use the $top and $skip query parameters. These parameters allow you to control the number of items returned and the starting point of the results.

For the Pipelines_ListByFactory API and other similar APIs, you can use these parameters as follows:

1. $top: Specifies the maximum number of items to return in the response.
2. $skip: Specifies the number of items to skip before starting to return results.

Here's how you can modify your API calls to include pagination:

1. Pipelines - List By Factory:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelines?api-version=2018-06-01&$top=50&$skip=0

2. Datasets - List By Factory:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/datasets?api-version=2018-06-01&$top=50&$skip=0

3. Data Flows - List By Factory:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/dataflows?api-version=2018-06-01&$top=50&$skip=0

4. Linked Services - List By Factory:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/linkedservices?api-version=2018-06-01&$top=50&$skip=0

To retrieve all pipelines (or other resources) when there are more than 50, you'll need to make multiple API calls, incrementing the $skip parameter each time. Here's a pseudo-code example of how you might implement this:

def get_all_pipelines(factory_name):

    pipelines = []

    skip = 0

    top = 50

    while True:

        url = f"https://management.azure.com/subscriptions/...&$top={top}&$skip={skip}"

        response = make_api_call(url)

        new_pipelines = response.json()['value']

        pipelines.extend(new_pipelines)

        if len(new_pipelines) < top:

            break  # We've retrieved all pipelines

        skip += top  # Prepare for the next page

    return pipelines

Thank you for your quick response.

However, the approach you suggested to use $top and $skip parameters does not seem to be working. I have tested multiple scenarios by changing the values of $top and $skip, but the API still returns only 50 pipelines, regardless of the values provided. I have attached a screenshot for reference.

Can you help me understand why the pagination parameters might not be working and suggest an alternative method to retrieve all 65 pipelines from my Data Factory?

Hi Vegnesa, Savio ,

Kindly check the below blog post which covers the implementation for your scenario:

https://datakuity.com/2022/07/01/perform-pagination-in-azure-data-factory/

Hope it helps. Kindly let us know how it goes. Thankyou