AzureDF IfCondition in ForEach activity not acting as expected

BrianChan 20 Reputation points
2024-07-09T09:22:06.5033333+00:00

Hi all,

I am new to Azure and I am testing on data factory.

I have a container which stored three CSV files. I would like to build a pipeline that copy the files to different output container base on the filename.

User's image

I added a Get Metadata activity and connect it to a ForEach activityUser's image

In the ForEach activity, I first set the filename to a variable and pass the variable to if Condition

User's image

The if Condition test on the below condition:

@startswith(variables('filename'), 'emp')

And the true/false activity are setting another variable which display the target copy locationUser's image

User's image

I ran a debug and found that the three output are all "/dep/department.csv" while I am expecting "/emp/emp_1.csv", "/emp/emp_2.csv", "/dep/department.csv"

User's image

Please help me out, thank you in advance.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,153 questions
Accepted answer
  1. Aravind Nuthalapati 75 Reputation points Microsoft Employee
    2024-07-09T15:16:57.5266667+00:00

    There is a limitation in forEach activity if you use set variable inside of it.
    Please make sure you have follow this workaround mentioned in the document by disabling the parallel setting inside of forEach activity.

    https://video2.skills-academy.com/en-us/azure/data-factory/control-flow-for-each-activity#limitations-and-workarounds

    User's image

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 19,781 Reputation points
    2024-07-09T09:47:57.8633333+00:00

    It seems like the issue is with how the IfCondition activity is evaluating the condition within the ForEach loop in your pipeline.

    First step you need to verify that within the ForEach activity, you are correctly setting the filename variable. You can do this with a Set Variable activity:

    • Set Variable activity:
      • Name: SetFilename
      • Variable: filename
      • Value: @item().name (assuming name is the property that contains the filename)

    You can add a Set Variable activity or a logging mechanism to confirm that the filename is being set correctly. For example:

    • Set Variable activity:
      • Name: LogFilename
      • Variable: log
      • Value: @variables('filename')

    I would go for setting up your IfCondition like below:

    • If Condition activity:
      • Condition: @startswith(variables('filename'), 'emp')
      • True activities: Set variable to /emp/ location
      • False activities: Set variable to /dep/ location