Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,567 questions
1 answer
One of the answers was accepted by the question author.
Data Factory Salesforce objects not the same as Objects seen in Salesforce URL
I have created a LinkedService to connect to the company supplied Salesforce URL using the customised url, the supplied user id, password and have acquired the correct security token. This connection tests successfully. It is also the same parameters…
asked 2020-07-03T04:14:49.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(80, 10%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
Binway
696
Reputation points
commented 2020-07-07T19:24:18.51+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
KranthiPakala-MSFT
46,462
Reputation points Microsoft Employee
0 answers
Spark Connector in ADF
Hi, I have created a spark connector to connect to azure data bricks. In copy activity source is spark connector and sink is Azure SQL DB. In spark Connector query, CreatedDate is being converted to String and throwing error where as it is timestamp…
asked 2020-06-19T16:53:27.433+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(64, 74%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Mounica
1
Reputation point
commented 2020-07-07T18:22:07.117+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
2 answers
Copy Activity data socurce preview error
Hi Team, Currently i have setup some tables move to dataLake using copy Activity pipeline, However i have realized few of tales have a issue when i connect it to copy activity source, According to error pop up as a below, Could you please check and…
asked 2020-06-19T14:15:06.213+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 19%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
Mahesh Madhusanka
216
Reputation points
commented 2020-07-07T18:19:19.377+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to checke pipeline cpu usage
Hi team , I have some pipelines. Whenever some pipelines run at that time cpu usage goes beyond 80. I have checked for cpu usage in task manager it is showing self-hosted ir process is running there which is taking too much cpu.. How can I determine…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(310.4, 98%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ER%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Error while creating data factory
I have free subscription account and i m trying to create data factory using sandbox and getting error 'deny' Policy action.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 79%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
Data Factory Variable Comparison not working as expected.
Hi All, I'm running a simple pipeline that takes an input variable, adds 1 day to it, and compares it to another variable. However, the comparison isn't working as I'm expecting it to. Variable boolTest is set to false after the update of the…
asked 2020-07-06T00:42:02.493+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(294.40000000000003, 21%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Robert Barat (WOW)
26
Reputation points
commented 2020-07-06T07:32:43.667+00:00
Robert Barat (WOW)
26
Reputation points
1 answer
One of the answers was accepted by the question author.
merge rows of same file Azure data factory
Hello I need to merge multiple rows into one row in azure data factory, per example I have the following file PolicyId Driver_name 0001 Adam 0001 Lucy 0002 peter At the end I need to have PolicyId Driver_name1 Driver name2 …
asked 2020-07-03T13:02:35.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 71%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECG%3C/text%3E%3C/svg%3E)
CORONADO GRANADOS Diana Milena
126
Reputation points
commented 2020-07-05T15:25:16.64+00:00
CORONADO GRANADOS Diana Milena
126
Reputation points
1 answer
One of the answers was accepted by the question author.
How to pull the data from Dynamic 365 to Data lake
<p>Hi <a rel="user" nodeId="74270" href="/answers/users/74270/chiragmishramsft-1092.html">@ChiragMishraMSFT-1092</a> </p> <p>Can you pls help me with the below error:</p> <p>{…
asked 2020-06-30T06:16:02.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2020-07-01T20:59:37+00:00
KranthiPakala-MSFT
46,462
Reputation points Microsoft Employee
1 answer
ADF - export to flat file with multiple record types
How to export to flat file with multiple record types in Azure Data Factory from JSON Example output file: Header, 1, H1 Detail1, 1, D11, Test1, 10 Detail1, 2, D12, Test2, 20 Detail2, 1, D21, 100 Detail2, 2, D22, 200 Trailer, 1, T1, 2,…
asked 2020-06-09T19:46:40.607+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 6%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sam
1
Reputation point
commented 2020-07-01T10:55:23.493+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 76%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
ChiragMishra-MSFT
956
Reputation points
1 answer
ADFV2 Copy Activity Support for more encodingNames
HI I am using the Copyactivity which generate the delimiter text files reading from parquet and applying the respective encodings for the files, we see a list of encoding which are supported by Copy activity (Delimited text format ), and still some…
asked 2020-06-30T22:36:06.483+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 88%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
ManojMathe
1
Reputation point
answered 2020-07-01T09:58:34.26+00:00
ChiragMishra-MSFT
956
Reputation points
2 answers
Azure Data Factory add Tags in json files
Hello, In the Portal it is possible to add Tags to a Azure Data Factory (I'm not talking about the Annotation in pipelines). But I can't find a way to include them in the ARM Template. I just created a blank ADF, added a Tag and exported the ARM…
asked 2020-06-24T09:24:10.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 51%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHL%3C/text%3E%3C/svg%3E)
Harry Leboeuf
6
Reputation points
answered 2020-07-01T06:39:43.22+00:00
Vaibhav Chaudhari
38,721
Reputation points
2 answers
Data flow error: 4501 - related to WranglingDataFlow
Hi guys, Please, for some reason, I'm getting this error message when I try to debug a pipeline with a wrangling data flow activity. { "errorCode": "4501", "message": "Failed to fetch…
asked 2020-06-04T13:26:01.637+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 3%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Michael S
1
Reputation point
commented 2020-07-01T01:55:45.807+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 52%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Ashutosh Shukla
1
Reputation point
1 answer
One of the answers was accepted by the question author.
JSON file Node didnt loade in SQL server tables using Azure data factory
HI Team, Below is MY JSON file for SOurce(Blob storage JSON files) I have loading Sql server table using Azure data factory(Copy data activity) in the Mapping tab they Node checkbox I have selected resolutions Checkbox data loaded only…
asked 2020-06-19T12:34:25.987+00:00
Anonymous
accepted 2020-06-30T07:03:03.28+00:00
Anonymous
2 answers
Log shipping
Dear experts I have two node SQL server standard failover cluster. I need to setup single node DR using log Shipping method. Want to ask after switch over to DR how we can setup, plan so that application can connect automatically to DR database…
asked 2020-06-29T05:40:19.84+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(230.39999999999998, 62%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
mukki mehra
1
Reputation point
answered 2020-06-29T12:55:40.187+00:00
Ronen Ariely
15,191
Reputation points
2 answers
One of the answers was accepted by the question author.
Azure Data Factory Dynamic Content
Hi there, I want to use a variable inside my Data factory Copy activity. I trying to use a variable which I got from an Azure Function. When I run the debug for the variable I can see that I'm getting the value but I can't access it as a…
asked 2020-06-27T10:33:26.767+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(275.2, 93%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ETE%3C/text%3E%3C/svg%3E)
Tahmid Eshayat
286
Reputation points
commented 2020-06-27T14:00:15.667+00:00
Tahmid Eshayat
286
Reputation points
1 answer
One of the answers was accepted by the question author.
Azure Data Factory Data Copy from Cosmos
We have one scenario, where we have to transfer/copy newly added/updated data from cosmos dB to a SQL Server every 2 hours. Currently, data copy copies all the files, can I configure to only copy the newly added/updated ? Thanks
asked 2020-06-26T03:57:27.08+00:00
Tahmid Eshayat
286
Reputation points
commented 2020-06-27T13:57:22.137+00:00
Tahmid Eshayat
286
Reputation points
1 answer
Data feild comma data impact to csv file creation issue
Hi Team, Currently i have setup a copy activity for a SQL query output .csv file create on the Data Lake destination, But in the one table address level details there have a comma (,) on the inside of the data fields, Due to the that reason, created…
asked 2020-06-19T17:49:07.423+00:00
Mahesh Madhusanka
216
Reputation points
commented 2020-06-26T17:54:55.45+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
1 answer
Doubt with new Azure IR
Hi there. I know this is maybe a trivial question for pretty much all of you, but I am confused with this. To reduce the Pipeline runtime, the administrator has created a new Azure IR. I understand that I need to change the setting on its…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 71%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
0 answers
data bricks scala : data frame column endoing from UTF 8 to windows 1252
HI I am working with data bricks where i have the data in parque and i am generating smaller files out of it , i have a column in this which is string and it has different characters and i have to encode this string value to windows 1252 or windows…
asked 2020-06-22T20:29:26.43+00:00
ManojMathe
1
Reputation point
commented 2020-06-25T22:21:25.563+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
how to replicate ongoing changes of on prem database to data lake?
Like AWS DMS handles ongoing replication and update bucket with respective files, what is available in azure to manage ongoing changes occurring at on-prem sql database? Initially it requires to do a full load in my data lake storage which can be…
asked 2020-06-19T17:52:50.293+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 42%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Seuli Das Majumder
101
Reputation points
accepted 2020-06-25T13:31:24.097+00:00
Seuli Das Majumder
101
Reputation points