Fetch multiple files from a specified folder (at once) from Azure Data Lake Gen2 (Preference1-API call, Prefercence2-Alternate options)

Amala Paulose 1 Reputation point
2022-03-04T12:38:30.52+00:00

We have a requirement to fetch multiple files in a specified folder from Azure Data Lake Gen2.

We have tried options specified in this article
"https://video2.skills-academy.com/en-us/rest/api/storageservices/datalakestoragegen2/path/read"
But it was not successful.

We are currently using Boomi HTTP connection to achieve this requirement.
Using wildcards in the URL/filename was not supported in Boomi HTTP.

However, we are able to retrieve single file at once by specifying the full URL (with exact file name).

Can you please suggest how to achieve fetching of files in bulk.
We are open to other alternatives as well.

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,466 questions
Azure Data Lake Analytics

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 32,906 Reputation points Microsoft Employee
    2022-03-15T14:42:44.48+00:00

    Hi @Amala Paulose ,

    Welcome to Microsoft Q&A platform and thanks for posting your query

    As I understand your query here, you want to fetch multiple files in bulk from a folder in Azure Data lake Gen2.
    As I tried to do some research , I didn't find any REST API to perform this action in bulk. However, using Path - List will be helpful to get the file list recursively. Post that you can you can loop through each files and try Path - Read to fetch each files.

    As you mentioned Boomi HTTP connector doesn't have option to perform wildcard operations, I will check this internally with our product team if there is anything that can be used as better approach.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.