Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions
1 answer
Data Factory Loading multiple Files. Ignore if column doesnt exist
Im loading multiple csv files using a wildcard file path and i have created parameters for the fileSystem, Directory and Filename (the csvs are in a datalake) A few of them have a couple of columns missing. is there any way in the copy activity to…
asked 2020-11-05T12:54:09.22+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
Debbie Edwards
521
Reputation points
answered 2020-11-06T23:34:27.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
Link services giving connection failed message
When trying to test connection on a linked service I get the error: Connection failed Failed to get response from server, please check network connection and retry. Contact support or send feedback if the issue persists. Activity ID:…
asked 2020-11-04T16:12:41.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 16%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMH%3C/text%3E%3C/svg%3E)
Matt Harper
1
Reputation point
commented 2020-11-06T21:24:22.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,801
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
RestAPI to Datalake storage
Hello guys, I am trying to pull data from the open sample API http://dummy.restapiexample.com/api/v1/employees And trying to load the data form it to Data Lake storage Account, but when the pipeline is debugged successfully, and when I…
asked 2020-11-05T13:16:13.343+00:00
Nimai Ahluwalia
41
Reputation points
accepted 2020-11-06T10:43:10.903+00:00
Nimai Ahluwalia
41
Reputation points
1 answer
One of the answers was accepted by the question author.
Use ADF to load a file in TEAMS into ADL
Is there a way to load an xlsx file in teams into Azure Data Lake using Azure Data Factory? Thanks geert
Azure Data Lake Storage
Azure Data Lake Storage
An Azure service that provides an enterprise-wide hyper-scale repository for big data analytic workloads and is integrated with Azure Blob Storage.
1,463 questions
Microsoft Teams
Microsoft Teams
A Microsoft customizable chat-based workspace.
9,954 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions
asked 2020-11-05T17:56:04.897+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 37%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EGV%3C/text%3E%3C/svg%3E)
geert vanhove
121
Reputation points
accepted 2020-11-06T09:14:21.173+00:00
geert vanhove
121
Reputation points
1 answer
Azure Data Factory Copy Activity is stuck in "Queued" status
Hello, last week i have created simple Copy Activity data Pipeline to move data from On Premise SQL to Azure SQL. Everything worked fine and took between 10 and 20 seconds to execute. Unfortunately this week when i execute pipeline, status is shown as…
asked 2020-10-29T14:54:49.107+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 65%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
andrucha
1
Reputation point
commented 2020-11-06T05:49:15.523+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
ADF 2011 Pipeline Error
I'm trying to run an SSIS package in ADF via the Execute SSIS Package activity. We are using the SSIS-IR setup with a self-hosted proxy using a self hosted IR and Azure Files. The self hosted IR is able to run normal ADF pipelines. Whenever I try to run…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 0%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
2 answers
FTPS issues again - 502 Command Not implemented
Hi all, I am getting an error trying to connect to yet another FTPS server.. Here is the exact error (Typo was in the error message): Failed to connect to FTP server. Please make sure the provided server informantion is correct, and try again. The…
asked 2020-10-06T20:33:46.99+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 45%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
John Aherne
516
Reputation points
answered 2020-11-05T22:23:50.567+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 71%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Akshay Mahajan
21
Reputation points
1 answer
Azure Data Factory key column watermakr workflow?
How to Incrementally copy data using Identity column or incrementing key column instead of time-stamp. We checked many blogs but didn't find how to implement this. This mechanisms is going to be used for ‘N’ # of tables. How can we implement…
asked 2020-10-29T16:20:43.833+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 35%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Anmol Ganju
1
Reputation point
commented 2020-11-05T17:37:19.427+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure Data Factory Workflow stopped working in automatic mode when started via trigger
Hi, My pipeline includes workflow process and running via trigger, then start every night. But the last two days ago, pipeline crashed. I am getting the following error: "Operation on target DWH_FACT_GeneralLadger failed:…
asked 2020-11-05T05:36:28.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 36%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYM%3C/text%3E%3C/svg%3E)
Yachnik Mykhailo
101
Reputation points
commented 2020-11-05T14:40:37.887+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
0 answers
Copy Activity Error when copying from Oracle to ASQL
Hi, I have been trying to do a simple copy activity from Oracle to Azure SQL Server. The source script is 'select * from Tableschema.TableName' and the target is the table name. The Oracle table contains over 4.5 million records. The copy…
asked 2020-10-31T13:46:28.583+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 94%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Alina Vembil
1
Reputation point
commented 2020-11-05T08:28:28.697+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Issues while fetching data from cosmos db using data factory
Hi All, I have been trying to fetch data from cosmos Db using a query which works well in cosmos Db explorer in Azure portal. But, the same query gives error while using it with Azure Data Factory. Query: SELECT * FROM c where…
asked 2020-11-03T18:58:03.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 25%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Manish Patel
31
Reputation points
commented 2020-11-05T08:23:47.47+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Copy data vs Data flow for cost
Hi Team, I am new In azure data factory what different between Copy data vs Data flow As my understand Copy data free (No Billing) Data flow is Billing each activity Please anybody help me. Thanks Shanvitha
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 79%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
ADF Internal Server Error.
In Azure Data Factory I am getting 'Internal Server Errors' when testing the connection of Linked Services and setting up Datasets. I get these errors in my own tenant as well as the tenant for the client I am working for at the moment. Both are located…
asked 2020-11-03T08:01:57.567+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 4%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJH%3C/text%3E%3C/svg%3E)
Jacques Herman
26
Reputation points
commented 2020-11-04T20:11:28.903+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
2 answers
Immediate restartability on failure of execute pipeline activity
There is a requirement to rerun the master peipline from the failed execute pipeline activity and its dependent tier pipelines. Though the existing capability helps to rerun the activities from the failed point but this only could be done after the…
asked 2020-10-30T09:39:22.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 57.99999999999999%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAO%3C/text%3E%3C/svg%3E)
Ankush O Wathodkar
21
Reputation points
commented 2020-11-04T19:02:36.177+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
One of the answers was accepted by the question author.
Power Shell Script Give Errors, While Stopping Triggers at Azure Data Factory
Hi, I am creating a release pipeline for the Azure Data Factory and My Data Factory Contains Triggers . Now I As we know that Before DataFacctory Release Deployment we have to Disable running triggers and we can use Azure Power shell script for that …
asked 2020-11-03T12:32:17.25+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 28.999999999999996%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
v-Shubham Kumar Jain
21
Reputation points
commented 2020-11-04T18:34:58.177+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
User configuration issue in azure pipeline in data factory
Hi, one of my users is facing below issue i am not able to find out why he is getting such error .. error code : 2402 Failure type : user configuration issue details : execution failed against sql server with error number 15165 .. could not…
asked 2020-11-02T07:43:01.883+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 28.000000000000004%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Rohit Boddu
466
Reputation points
commented 2020-11-04T13:18:29.67+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Updating RU value of a container in Cosmos DB from ADF pipeline
Hi, I'm just curious to know whether anyone has tried updating the RU value of a container in Cosmos DB account using ADF pipeline. If yes, please share the details.
asked 2020-11-02T22:51:37.757+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 9%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBN%3C/text%3E%3C/svg%3E)
bharath nunepalli
41
Reputation points
commented 2020-11-04T12:18:00.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 10%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KalyanChanumolu-MSFT
8,321
Reputation points
2 answers
Error while loading data from Azure Data Lake Gen2 to Azure Synapse Analytics using Azure Data Factory
Hi All, I am getting an error while loading data from Azure Data Lake Gen2 to Azure Synapse Analytics using Azure Data Factory. I am unable what exactly the error is? What permission has to be provided? How?. This error occurs when I use Polybase…
asked 2020-08-16T09:00:27.563+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 55.00000000000001%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EI%3C/text%3E%3C/svg%3E)
ImranMaqsood
6
Reputation points
answered 2020-11-04T04:57:11.47+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(115.19999999999999, 77%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERH%3C/text%3E%3C/svg%3E)
Rudolph Harms
26
Reputation points
3 answers
One of the answers was accepted by the question author.
Load CSV to ADLS by Date
I have a large CSV file that contains a extract of records dating back to year 2016. The requirement is to copy the data into azure data lake but in year/month/day.csv format. What is the best approach to implement this? I am aware of how to achieve the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 88%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
1 answer
Azure Blob Storage Dataset Wild card file name
Hello, I have a requirement where in the user will upload delimited file in the Azure Blob Storage and the Azure Data Factory pipeline will copy the file from Azure Blob Storage and dump it into Azure SQL DB. To achieve this, I have created…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)