Parameters for Data Factory in Microsoft Fabric

This document describes how to use parameters in your pipelines for Data Factory in Fabric.

How to use parameters, expressions and functions in pipelines for Data Factory in Fabric

In this document, we focus on learning fundamental concepts with various examples to explore the ability to create parameterized data pipelines within Data Factory in Fabric. Parameterization and dynamic expressions can save a tremendous amount of time and allow for a much more flexible Extract, Transform, Load (ETL) or Extract, Load, Transform (ELT) solution, which will dramatically reduce the cost of solution maintenance and speed up the implementation of new features into existing pipelines. These gains are because parameterization minimizes the amount of hard coding and increases the number of reusable objects and processes in a solution.

Parameter and expression concepts

You can use parameters to pass external values into pipelines. Once the parameter has been passed into the resource, it cannot be changed. By parameterizing resources, you can reuse them with different values each time. Parameters can be used individually or as a part of expressions. Parameter values in the definition can be literal or expressions that are evaluated at runtime.

Expressions can appear anywhere in a string value and always generate another string value. Here, password is a pipeline parameter in the expression. If a parameter value is an expression, the body of the expression is extracted by removing the at-sign (@). If a literal string is needed that starts with @, it must be escaped by using @@. The following examples show how expressions are evaluated.

Expressions can also appear inside strings, using a feature called string interpolation where expressions are wrapped in @{ ... }. For example, the following string includes parameter values and literal string values:

"First Name: @{pipeline().parameters.firstName} Last Name: @{pipeline().parameters.lastName}"

Using string interpolation, the result is always a string. For example, if you defined myNumber as 42 and myString as foo:

Examples of using parameters in expressions

Creating and using parameters

To create parameters, select the background of the pipeline editor canvas, and then the Parameters tab of the properties window at the bottom. Select the + New button to add a new parameter to the pipeline, give it a name, a data type, and a default value:

You can then use the parameter anywhere in your pipeline where dynamic content is supported. In this example, the parameter is used to dynamically provide the name of a Lakehouse data store on the Source tab of a copy activity's property pages.

The Add dynamic content window is displayed, allowing you to specify any kind of dynamic content, including parameters, system variables, functions, or pipeline variables. In this example, the previously defined parameter is selected, and the dynamic content window is automatically populated with the correct expression to reference the parameter.

Complex expression example

The below example shows a complex example that references a deep sub-field of activity output. To reference a pipeline parameter that evaluates to a sub-field, use [] syntax instead of dot(.) operator (as in case of subfield1 and subfield2)

@activity('*activityName*').output.*subfield1*.*subfield2*[pipeline().parameters.*subfield3*].*subfield4*

Dynamic content editor

The dynamic content editor automatically escapes characters in your content when you finish editing. For example, the following content in the content editor is a string interpolation with an expression function:

@{toUpper('myData')}

The dynamic content editor converts the above content to the following expression:

MYDATA

Using functions and variables in expressions

You can call functions and use variables within expressions. The following sections provide information about the functions that can be used in an expression.

Pipeline scope variables

These system variables can be referenced anywhere in the pipeline JSON.

String functions

To work with strings, you can use these string functions and also some collection functions. String functions work only on strings.

Collection functions

To work with collections, generally arrays, strings, and sometimes, dictionaries, you can use these collection functions.

Logical functions

These functions are useful inside conditions, they can be used to evaluate any type of logic.

Conversion functions

These functions are used to convert between each of the native types in the language:

• string
• integer
• float
• boolean
• arrays
• dictionaries

Math functions

These functions can be used for either types of numbers: integers and floats.

Date functions

Related content

• Expression language