How can I automatically delete the files from the nodes of Azure dwpool

Sijo Johny 21 Reputation points
2022-12-01T12:45:05.993+00:00

Hi ,

I have set "RetentionTimeInDays" of azure batch in Azure Data Factory to 0, seeing the answer in this site. But seems like its not working. I can see the files in the Node wd directory even after one day. Is there are any other way to delete the files from node automatically. The reason for deletion is in certain times the node become unusable because of the disk size getting full

Regards
Sijo Johny

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
330 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,681 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 38,441 Reputation points Microsoft Employee
    2022-12-02T09:39:28.13+00:00

    Hi @Sijo Johny ,

    Thank you for posting query in Microsoft Q&A Platform.

    If I understand correctly, you are using custom activity here to run some script on batch pool. Please correct me if I am wrong.

    RetentionTimeInDays setting is actually, the retention time for the files submitted for custom activity. Default value is 30 days.

    If you are looking of deleting the script files which submitted to batch pool from storage account, then you can consider using Delete activity and configuring dataset to corresponding storage location.

    Check below videos to understand delete activity better.
    Delete Activity in Azure Data Factory
    Delete old files from storage using Azure Data Factory

    If you are looking for deleting some files from bath pool nodes then consider writing python or PowerShell script and run that script on bath pool.

    Hope this helps. Please let me know if any further queries.

    Please consider hitting Accept Answer button. Accepted answers help community as well.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.