Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,016 questions
0 answers
ADF copy activity for REST API POST requiring string value in request body field
I have a ADF copy activity where my source is a REST API POST statement to a report process in one of our data sources. I have the parameters for the report stored in a SQL server table and I am calling it from a pipeline parameter. My expression for the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(19.2, 57.99999999999999%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECC%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How much would it cost to create an ADF Pipeline with two servers
I need to retrieve data from a MySQL backup file on Cyberduck, restore it in MySQL Azure then move it to our data warehouse in SQL Azure, how much would it cost for building the pipeline using Azure ADF.
asked 2024-07-03T14:38:13.8133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 97%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYN%3C/text%3E%3C/svg%3E)
Youness Naciri
20
Reputation points
accepted 2024-07-05T19:37:49.6533333+00:00
Youness Naciri
20
Reputation points
3 answers
SQL Bulk Copy failed
SQL Bulk Copy failed due to receive an invalid column length from the bcp client. I am getting this error while copying data from adls to sqlserver. I am copying a csv file. I checked the length of each and every column. Destination has more length when…
asked 2022-09-26T05:37:18.833+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 89%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKJ%3C/text%3E%3C/svg%3E)
KEERTHANA JAYADEVAN
66
Reputation points
answered 2024-07-05T18:15:52.4+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 70%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
ravi555
0
Reputation points
1 answer
Alternative to keyValues Function for Pipeline Expressions in ADF
I'm working with Azure Data Factory (ADF) and I need to create a key-value map within a pipeline expression. While the keyValues function works perfectly in Data Flows, it's not available in pipeline expressions. My scenario involves two separate arrays,…
asked 2024-07-05T13:03:46.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 32%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
Glasier
360
Reputation points
answered 2024-07-05T16:47:29.7433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,100
Reputation points Microsoft Vendor
1 answer
Error Occurred While Importing Test Cases - Test Case IDs Not Found
Description: We encountered an error while importing test cases into the second Azure DevOps board. The error message received is as follows: Error occurred while importing one or more test cases: Test case(s) IDs not found in the current suite. Please…
asked 2024-07-05T12:21:47.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 46%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESG%3C/text%3E%3C/svg%3E)
Sathishkumar G
0
Reputation points
answered 2024-07-05T16:43:28.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 32%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
Chidiebere_Ugwu
0
Reputation points
1 answer
To Deploy Azure ARM template which is greater than 4MB from one environment to another environment
Hi, I was trying to move a Azure data factory from DEV to QA using ARM template which has a size of 20 MB. When i try to export the ARM template from DEV data factory and try to import it in QA data factory using Azure custom deployment, It fails to…
asked 2024-07-02T10:13:45.4633333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(195.2, 56.00000000000001%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESB%3C/text%3E%3C/svg%3E)
Srihari B
20
Reputation points
commented 2024-07-05T16:02:56.95+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,616
Reputation points Microsoft Employee
1 answer
How to fix "out of memory exception " while processing a pipeline of around 43 gb of data using copy activity?
I am processing a pipeline of around 43gb of data using copy activity and i am getting the error as : ErrorCode=SystemErrorOutOfMemory,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A task failed with out of…
asked 2024-06-24T10:25:41.07+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(140.8, 17%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELM%3C/text%3E%3C/svg%3E)
Lakshmi Moulya Nerella
0
Reputation points
commented 2024-07-05T15:59:58.39+00:00
BhargavaGunnam-MSFT
28,616
Reputation points Microsoft Employee
0 answers
Why are so many Workflow Orchestration Manager configurations locked? Especially AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION is annoying.
What is the reason that so many configuration settings are locked, some to non-default values? For example, we would like to turn AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION to True but it is currently locked to False. This greatly hinders our workflow:…
asked 2024-07-03T10:45:02.1733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 19%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
aeteq
0
Reputation points
commented 2024-07-05T15:40:09.9466667+00:00
BhargavaGunnam-MSFT
28,616
Reputation points Microsoft Employee
3 answers
How do you stop/pause or restart a Workflow Orchestration Manager instance?
We use git-sync with our WOM instance. For DAGs, this works fine, changes to DAGs show up very quickly in Airflow. However, some changes require a restart of the instance, for example changes in the plugin (since those are only loaded once at startup).…
asked 2024-07-03T10:50:27.2933333+00:00
aeteq
0
Reputation points
edited an answer 2024-07-05T15:22:53.2166667+00:00
BhargavaGunnam-MSFT
28,616
Reputation points Microsoft Employee
0 answers
Copy CSV files to citrix sharefile path (doman.sharefile.com) folder using ADF
Hello, I am trying to do a POC, which is to build a small pipeline and I am trying to export a SQL table (AZ SQL DB) to a CSV file and need to upload it on citrix sharefile path, site url looks like "https://domain.sharefile.com/home/shared".…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 1%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
Creating linked service in azure data factory to connect azure storage account
Hi, I have created azure data factory with out "Managed Virtual Network" . Created azure storage account and disabled 'Public network access' in storage account, Both data factory and storage accounts are in same vnet and same subnet. Later, I…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 37%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
1 answer
How do you add Airflow requirements to a Workflow Orchestration Manager instance without it crashing?
The Workflow Orchestration Manager has an option to add pypi dependencies. However, when I add for example "apache-airflow-providers-databricks", the instance does not start anymore (it tries to start for ~1 hour and then stops with an…
1 answer
Is IMDS available on Azure-SSIS VMs?
Is the Azure Instance Metadata Service (IMDS) (169.254.169.254) available in an ADF Azure-SSIS integration runtime? If it isn't available, how else can I get a bearer token? eg. to be able access Azure key vaults. I am looking to find alternatives to not…
asked 2024-07-05T04:42:11.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(22.400000000000002, 64%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDJ%3C/text%3E%3C/svg%3E)
Daniel Jarolim
0
Reputation points
answered 2024-07-05T12:30:04.4233333+00:00
PRADEEPCHEEKATLA-MSFT
84,381
Reputation points Microsoft Employee
0 answers
I wan use Show billing report by pipeline in ADF, and enabled this via CI/CD
I want enabled the feature "Show billing report by pipeline " in azure Data Factory, but I don't know how can I enabled this via Terraform deployment or ARM template deployment. How Can I do that ?
asked 2024-07-03T15:15:45.9433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 9%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBB%3C/text%3E%3C/svg%3E)
Benjamin Baele
0
Reputation points Microsoft Employee
commented 2024-07-05T12:20:37.4333333+00:00
Benjamin Baele
0
Reputation points Microsoft Employee
1 answer
How to create sha256 signature for Oracle Cloud integration Rest API ?
Hi Experts, Infra: Azure Synapse -we are using pipelines / API method to get data and create a Data Warehouse. Source: We have an Oracle Cloud Infrastructure (OCI ) as source , that will output the report to Oracle Cloud Storage. OCI has a REST API…
asked 2024-07-04T03:27:38.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Caniut Alex
0
Reputation points
commented 2024-07-05T11:29:07.3166667+00:00
Smaran Thoomu
12,100
Reputation points Microsoft Vendor
0 answers
Using powershell, how can I set in a data factory a shared integration runtime which exists in a different subscription?
Hi team, I'm looking to configure a shared integration runtime in a ADF which resides in a different subscription were the IR exists. I reviewed…
asked 2024-07-02T22:57:14.5366667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 10%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERM%3C/text%3E%3C/svg%3E)
Reyes Monsalve, Ruben Dario
26
Reputation points
commented 2024-07-05T11:26:40.41+00:00
Smaran Thoomu
12,100
Reputation points Microsoft Vendor
0 answers
When will Workflow Orchestration Manager be updated to newer Airflow versions?
The WOM has been stuck at Airflow Version 2.6.3 for a while now. The current upstream Airflow version is 2.9.2. Is there any (public) timeline on when newer versions will be available? Is there a general release schedule?
0 answers
How can you import data into the Workflow Orchestration Manager Database?
We have been using Airflow on premise for a long time and would like to migrate to WOM without losing our execution history. Is there a way to import data into the Airflow Meta-DB?
2 answers
IF Condition with 'AND' and 'OR' Operator in ADF Pipeline Expression Builder
I am making an API call in Azure Function and retrieving values for 7 diff attributes. Out of these 7 attributes 6 of the attributes return Boolean value,whereas one attribute Requested Timestamp will either return Null or some timestamp value. I am…
asked 2023-11-07T16:00:42.4533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(32, 40%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Anubhav Sinha
0
Reputation points
commented 2024-07-05T10:45:09.4866667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 57.99999999999999%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Mike Langlois
10
Reputation points
0 answers
The new snowflake connector is by default altering the session to set multi_statement_count = 1
I was using script activity to execute multiple statements in Snowflake using ADF. The new snowflake connector is not able to run Snowsql statements in one script activity separated by semicolon. In the Warehouse, when I check for query history, below…
asked 2024-07-03T05:22:14.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 13%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERG%3C/text%3E%3C/svg%3E)
Raghu Gorla
0
Reputation points
commented 2024-07-05T09:52:04.9233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
8,080
Reputation points Microsoft Vendor