How do I unzip files in a blob from the command in Batch Service?

Anshul Rawat 0 Reputation points
2023-01-23T13:54:41.7666667+00:00

Hello,

I am trying to run a python script from my batch service activity in ADF as follows:

cmd /C python main.py

However, this runs into an error since there are multiple files to be processed by the python script which leads to error code: 2504 [The total size of resourceFiles can't be more than 32,768 characters] ([https://video2.skills-academy.com/en-us/azure/data-factory/data-factory-troubleshoot-guide#error-code-2504)

As a workaround, the documentation suggests to upload a zipped version of all the files and then unzip it in the command as follows:

powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory($zipFile, $folder); }" ;

but this does not work for me and the pipeline gets stuck in processing. I don't understand why this is happening. Is this related to the type of access [admin/nonadmin] of ADF on the batch pool? Either way, please suggest a way of unzipping this file from the batch service activity.

Azure Batch
Azure Batch
An Azure service that provides cloud-scale job scheduling and compute management.
330 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,681 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Samy Abdul 3,371 Reputation points
    2023-01-23T14:24:04.7+00:00

    Hi @Anshul Rawat , the limitations of zipping and unzipping files for batch has been discussed below:

    Zipping and UnZipping files for Batch limitation in ADF

    https://video2.skills-academy.com/en-us/answers/questions/227787/zipping-and-unzipping-files-for-batch-limitation-i Thanks

  2. AnnuKumari-MSFT 32,906 Reputation points Microsoft Employee
    2023-01-30T04:50:55.0966667+00:00

    Hi Anshul Rawat ,

    Welcome to Microsoft Q&A platform and thanks for posting your question here. As I understand your issue, you are facing an error: "The total size of resourceFiles can't be more than 32,768 characters" , while trying to unzip a file using custom activity. Please let me know if that is not the case.

    Kindly check if the below posts where other customers encountered similar issue has been mitigated , help you resolve the issue:

    Zipping and UnZipping files for Batch limitation in ADF

    ADF Custom Activity : Total size of resourceFiles cannot be more than 32768 characters

    Please let us know if it was helpful. Thankyou.

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.