Not able to copy entire files and folders recursively in specific folder in Azure Data Factory

Rohit Boddu 466

Hi,

I want to copy data between two adls gen2 accounts. I am unable to copy all files and subfolders in specific folder to the destination.
Please check the attached screenshot which shows directory structure of my source adls account and destination adls account.

In my source adls account i have container "source-container" which has folder named "sourcefolder1" in it and which contains few files and few directories. i used azure data factory to copy data and i used wildcard method to copy the data.

you can see in attached screenshot that i have 3 folders and some files in my source folder but after copying it has copied only files and not the directories.

previously using the same method i was able to copy all files and folders recursively but now it is failing.

In adf setting i have used "preserve hierachy" as copy behaviour and set "Recursive" to true in settings.

Regards,
Rohit Boddu.

John Aherne 516 Reputation points

2020-09-23T19:38:18.587+00:00

Do you have "Recursively" checked in your copy activity for the source?
Rohit Boddu 466 Reputation points

2020-09-24T03:47:49.717+00:00

yes @John Aherne i have already checked Recursively in source settings. But it is not copying entire folder and subfolder ..

Below is the screenshot of my source settings.
John Aherne 516 Reputation points

2020-09-24T17:54:13.657+00:00

One other setting to check would be on the Sink settings - For "Copy Behavior" do you have "Preserve Hierarchy" selected?
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2020-09-25T00:55:11.357+00:00
Firstly thank you so much @John Aherne for sharing your inputs and helping the community. :)

@Rohit Boddu , As @John Aherne called out, please refer to below documentation to explore about recursive and copyBehavior setting in copy activity.

recursive and copyBehavior examples

Hope this helps. Let us know how it goes.

Thank you
Rohit Boddu 466 Reputation points

2020-09-25T04:42:40.077+00:00

Yes @John Aherne i have also set copy behavior to "Preserve Hierarchy but its not copying recursively.

Is it possible that can we connect through teams so i can share my screen and show you the pipeline. If yes please let me know.

Regards,
Rohit .
Rohit Boddu 466 Reputation points

2020-09-25T05:11:35.767+00:00

yes @John Aherne i have Set copy-behavior as "Preserve Hierarchy" in pipeline but still it is not working.

Can we connect in teams so i can share my screen and show you the pipeline. If yes then you can reach me at rohit.m.boddu@harsh.com .com.

Regards,
Rohit
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2020-09-28T22:16:13.097+00:00
Hi @Rohit Boddu ,
Sorry for your experince. If you still experiencing the issue, could you please download and share your pipeline support files to AzCommunity[at]microsoft[dot]com with below details.

Email subject: <Attn - Kranthi : Microsoft Q&A Thread title>

Thread URL: <Microsoft Q&A Thread>

Subscription ID: <your subscription id>

Data Factory Region

Pipeline runID

Attached support files of your pipeline

Please let me know here once the email is sent.
Thank you
Rene Løhde 211 Reputation points

2020-09-30T08:44:05.507+00:00

@KranthiPakala-MSFT ...I have the exact same problem, would you like the same debugging info from me?
KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2020-10-01T01:05:05.36+00:00

Hi @Rene Løhde , Thanks for reaching out here. Sure, please do share the requested details as mentioned in my previous comment.

Thank you
Rene Løhde 211 Reputation points

2020-10-01T05:06:06.837+00:00

Thanks, email sent.

2 answers

KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2020-10-02T18:59:07.11+00:00

Hi @Rene Løhde ,

Thanks for your email. Could you please try using Wildcard file path setting in source settings of your copy as shown below:

Hi @Rohit Boddu - Could you please clear browser cache and restart your browser and try below wildcard format in your source settings of Copy and see if that helps.

I tried the above wildcard file path format and Copy activity is working as expected. Please let us know how it goes.
Looking forward to your confirmation.
Thank you
Please sign in to rate this answer.

1 person found this answer helpful.
Rene Løhde 211 Reputation points

2020-10-04T11:40:14.193+00:00

Hi @KranthiPakala-MSFT ,

Thank you for taking the time - that did the trick! Wildcarding directly after the Dir* was my problem.

KranthiPakala-MSFT 46,462 Reputation points Microsoft Employee

2020-10-05T18:21:29.33+00:00

Hi @Rene Løhde ,

Glad to know that the above suggestion was helpful to resolve your issue. Please feel free to up-vote the post that helped you resolve the issue so that others can be benefited by reading this post.

Thank you

----------

Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.

Rohit Boddu 466 Reputation points

2020-10-12T12:05:10.193+00:00

Hi @KranthiPakala-MSFT sorry i was on off for last 2 weeks so didnt see ur messages.

I am still having problem in copying recursively all files and folders .. as suggested i have cleared cache even though it is not working ..

actually i m trying to copy data from multiple tables in one go .. pipeline is copying data from the tables but it is skipping some of the folders in it.. please see attached screenshot

i will send the all the information that u have asked on mail.

Regards,
Rohit.

Rohit Boddu 466 Reputation points

2020-10-12T12:20:53.187+00:00

@KranthiPakala-MSFT i have sent mail with all needed information that u aksed ..

Rohit Boddu 466 Reputation points

2020-10-21T04:59:02.57+00:00

Hi @KranthiPakala-MSFT run pipeline as u suggested but still not able to copy emprty folders to destination.. if folder contains files then only it is getting copied ..

Thanks & Regards,
Rohit
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
MartinJaffer-MSFT 26,081 Reputation points

2020-10-20T23:29:17.613+00:00
My findings as of 2020-10-20:

If the folder is empty, it is not copied, unless you select at least one of the following settings:

Preserve ACL

Preserve Owner

Preserve Group

Any one , or combination of, these will cause the empty folders on ADLS gen2 to be copied over.
I am inquiring internally as to whether this is by design.
Please sign in to rate this answer.
MartinJaffer-MSFT 26,081 Reputation points

2020-10-23T17:57:42.163+00:00

@Rohit Boddu I found that it is by design that currently, only files are copied, save for the above method.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Not able to copy entire files and folders recursively in specific folder in Azure Data Factory

2 answers

Your answer