When using "Copy data" recursively to copy by last modified, what is the difference between "File path in dataset" vs "Wildcard file path"?

Vadim O 0 Reputation points
2025-11-04T13:24:41.04+00:00

A pipeline that copies files from Source to Destination runs every 2 hours and at one point few weeks ago it missed files in the source that were modified and not copied to the Destination. The files were in a subdirectory. When I reran the pipeline with specific start and end times the files copied over.

Only difference on pipeline that missed files is it has "File path type" set to "File path in dataset" while others use "Wildcard file path" option.

Shouldn't both produce the same result, and if now would it be the cause for modified files in subdirectory be missed? Examples below.

Screenshot 2025-11-04 081745

Screenshot 2025-11-04 081830

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
{count} votes

1 answer

Sort by: Most helpful
  1. Amira Bedhiafi 40,576 Reputation points Volunteer Moderator
    2025-11-04T18:32:29.9833333+00:00

    Hello Vadim !

    Thank you for posting on Microsoft Learn Q&A.

    The 1st file path in dataset you are pointing the dataset to one concrete folder and optionally one concrete file name so the Copy activity will list the files that are under that folder andonly if recursively is checked also under its subfolders and it does not evaluate wildcards here.

    The 2nd wildcard file path you can put */? in the folder part or the file name so ADF first expands the pattern and then applies the last modified filter to the matched files and it is useful to pick specific branches or file name patterns across many child folders.

    If the dataset version has file path in dataset with a file name filled in, ADF looks only for that exact file in the root folder and it won’t pick differently named files that sit in subfolders.

    With wildcard file path, ADF matches all files and then filters by last modified so those subfolder files get included.

    If you want to copy everything changed in the tree either you leave the dataset file name empty and keep recursively or use wildcard file path with file: * and if needed wildcards in the folder path to target certain branches.

    0 comments No comments

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.