I have a issue with data lake gen2 with multiple folders and i need to delete grathan 60 days files in gen with multiple folders

anji reddy 1 Reputation point
2022-06-08T04:17:58.843+00:00

209322-image.png

209170-image.png

209361-image.png

209342-image.png

Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,466 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,681 questions
Azure Data Lake Analytics
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Subashri Vasudevan 11,206 Reputation points
    2022-06-08T14:34:20.247+00:00

    Hi @anji reddy ,

    thanks for posting your query in MS Q&A portal.

    As per my understanding, you have a nested folder structure like folder1\folder2 and inside folder2 you have many files. And you want to delete files older than 60 days. Please let me know if my understanding is correct. (If you have a different nesting for folders, please let us know.)

    What you are trying to do is, have a Master pipeline and get the folder names, and loop through each folder using foreach loop and call a child pipeline. In child pipeline, you use get meta data activity to get file names greater than 60 days and then use a delete activity. You encounter an error when filtering files which are older than 60 days in get metadata activity.

    Your approach to loop through nested folders seems fine. In the child pipeline you can delete the files by following this video

    Also please check this link to filter files using delete activity itself

    Please check and let us know if you have further questions on it.

    thank you.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.