Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions
1 answer
One of the answers was accepted by the question author.
Not able to connect to blob storage from data factory using managed identity (in a pipeline)
I created a linked service for blob storage using managed identity and it works when I test connection. But when I put it to a pipeline, it gives an error. The data factory has access on the blob storage. Details discussed with @Anonymous and she…
asked 2020-11-18T04:05:10.463+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 6%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKR%3C/text%3E%3C/svg%3E)
Kothai Ramanathan
941
Reputation points Microsoft Employee
accepted 2020-12-02T04:37:58.533+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Data Factory Calling a Logic App
Hi, I am calling a logic app but I need the body of the HTTP call to be dynamic, not fixed. How can this be done? Thanks! Mike Kiser
asked 2020-11-23T16:37:20.7+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 49%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMK%3C/text%3E%3C/svg%3E)
Mike Kiser
1,536
Reputation points
commented 2020-12-01T19:53:35.29+00:00
Mike Kiser
1,536
Reputation points
0 answers
how to get Data Factory to connect to my Azure SQL database
I have a Azure SQL Database configured to be accessed from a Private Enpoint that lives in a VNet I configured. I'm trying to figure out how to get data factory to be able to connect to the database. Is this possible
asked 2020-11-12T15:21:02.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 18%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
JayakrishnaGunnam
1
Reputation point
commented 2020-12-01T19:12:23.587+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,801
Reputation points Microsoft Employee
0 answers
Create DataFlow using .Net SDK
Respected, I was able to create Data Factory and Pipeline with CopyActivity as mentioned in the quick tutorial. https://video2.skills-academy.com/en-us/azure/data-factory/quickstart-create-data-factory-dot-net I am not able to make it work when I…
asked 2020-11-13T03:47:07.217+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jigsm
236
Reputation points
commented 2020-12-01T19:10:10.213+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
How to optimize Azure Data FactoryPipeline?
Hi All, Need help on the below requirement... I have to perform ADF migration from 90(source and target Azure MI databases) For Example Source on-prem servers(data server1, data server2, data server3,...data server90) to Target Azure MI Servers(Azure MI…
asked 2020-11-30T19:07:13.233+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 8%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anonymous
commented 2020-12-01T15:18:15.527+00:00
Anonymous
2 answers
ADHOC Pipeline execution based on date range(startdate and endate)
Hi Team, I want to execute the pipeline based on start date and enddate and it should load the data into ADLS in the incremental way. for example start date=2020-11-14 and enddate=2020-11-17 then it should load the file into data lake like…
asked 2020-11-18T15:05:47.357+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 98%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPC%3C/text%3E%3C/svg%3E)
pankaj chaturvedi
86
Reputation points
commented 2020-12-01T12:24:44.4+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
1 answer
Maximum activities per pipeline, which includes inner activities for containers - 40. What does the container signify here?
An example to understand how the inner activity limit would work for the "container" would be very helpful. Want to understand the significance of container and its logical existence.
asked 2020-11-25T07:52:12.44+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 57.99999999999999%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAO%3C/text%3E%3C/svg%3E)
Ankush O Wathodkar
21
Reputation points
commented 2020-12-01T12:20:36.56+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
0 answers
Azure SSIS IR execution SSIS just hanged
Anyone experience the following situation ?
asked 2020-11-25T09:15:15.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 40%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
sakuraime
2,326
Reputation points
commented 2020-12-01T11:48:50.887+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Can I specificy the VPC for Datafactory
I have a site to site VPN setup and working. I can prove this by launching a virtual machine into the virtual network and pinging a machine in the other site. Now what I want to do is launch Data-factory in this virtual network so that it can connect…
asked 2020-11-27T22:08:46.417+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 8%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERD%3C/text%3E%3C/svg%3E)
Randall Danielson
96
Reputation points
accepted 2020-12-01T01:01:53.327+00:00
Randall Danielson
96
Reputation points
1 answer
One of the answers was accepted by the question author.
Can ADF V2 Data flow Derived column task alter same column in Mutiples flat files with one flow or task
Hello, I wanted to use Data flow derived column activity to alter same column name in multiple files , I ahve not seen any good tutorial or videos on that. I have column called Account ID in four .csv files under ADLS Gen 2 folder 'MYFolder/2020/11…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 38%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Datetime issue in Azure Data Factory
There is datetime conversion issue while loading data incrementally from SQL Server to Azure SQL server using timestamp. The datetime should be converted to datetime2 at source side. Please suggest any work around to this issue?
asked 2020-11-17T09:07:29.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(227.2, 42%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMD%3C/text%3E%3C/svg%3E)
Moulshree Dhanawade
1
Reputation point
answered 2020-11-30T22:12:43.37+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Azure Data Factory V2 - Add TriggerTime to Kusto query
Hi, I want to run Kusto query with a time parameter in ADF so I can debug conveniently. I try many times and want to know how to pass parameter to it although I can write 'let timestring = now()' to make code works. Here is part of my Kusto code. let…
asked 2020-11-25T15:51:35.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(208, 67%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
GouKong
126
Reputation points
commented 2020-11-30T10:55:20.527+00:00
PRADEEPCHEEKATLA-MSFT
88,716
Reputation points Microsoft Employee
3 answers
Rerun from the next activity in the Data Factory piepline
Lets assume that I have a pipeline with activities A->B->C->D If an activity B fails then I know that we can rerun from the failed activity B. Lets say if an activity B fails and I want to skip Activity B and run from Activity C. Can we do…
asked 2020-11-25T00:15:33.197+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 66%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
susheel
1
Reputation point
commented 2020-11-30T08:23:12.387+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
2 answers
Dynamic query to select all the columns except PK from multiple table using SQL Query
The below script is throwing an error of Duplicate StudentID column. We need SQL script so we can select all column except the PK column. We can't simple write those column names as the script is going to be dynamic. SELECT…
asked 2020-11-25T17:03:31.177+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 89%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESP%3C/text%3E%3C/svg%3E)
Sarvesh Pandey
71
Reputation points
commented 2020-11-30T01:43:51.727+00:00
MelissaMa-MSFT
24,196
Reputation points
1 answer
One of the answers was accepted by the question author.
data factory - databricks
Hello, I am in the process of learning the data factory. 1- Is there anything that data factory/data flow does not do which means it needs to be done in databricks? I am trying to find-out that if I know data factory really well, then is there a…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 97%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
Issue Flattening JSON file with paging to SQL with Data Flow
I'm using a REST API data source to create a JSON file and place it in Azure Blob Storage. Then I trying to flatten the file within the data flow and sink it to an SQL database. I keep getting an error when trying to sink because I cannot get the…
asked 2020-11-06T15:24:19.513+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAH%3C/text%3E%3C/svg%3E)
Andrew Horton
6
Reputation points
commented 2020-11-27T15:50:37.377+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Connecting from Logic apps to Data factory using Managed Identity
I am trying to trigger a pipeline run from a logic app. When I create a connection for Azure Data Factory (using service principal), I only see the option to connect via Client Id/Client secret and not via managed identity. Can we not connect from…
asked 2020-11-25T09:55:11.657+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
commented 2020-11-27T04:33:36.387+00:00
Kothai Ramanathan
941
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Import CSV with invalid header name into Sql Database using Azure Data Factory
I need to import a CSV file that has a reserved word as column name column name examples: date, day
asked 2020-11-24T18:12:06.707+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 5%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Carlos Montoro
21
Reputation points
commented 2020-11-27T01:14:27.5+00:00
Carlos Montoro
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Mapping values from a dataset into a derived column for another dataset
Hi everyone. Would like to understand if there is a way in data flow to map a value from one dataset into a derived column for another dataset. This is my use case: I have an excel workbook in which the first tab contains a field with the file's…
asked 2020-11-18T18:39:34.153+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 68%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
HansJKiamzon
56
Reputation points
accepted 2020-11-26T17:47:59.267+00:00
HansJKiamzon
56
Reputation points
1 answer
Copy from sftp fails but Output details shows all good
Hi I am copying files from SFTP Site to ADLS Gen2 , i want to create the file on the fly (do i need the create a file upfront in ADLS Gen 2?). When i debug in the details i get the following information But when i refresh the results it shows…
asked 2020-11-16T21:40:48.93+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 7.000000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
bk
466
Reputation points
commented 2020-11-26T15:43:55.81+00:00
HarithaMaddi-MSFT
10,136
Reputation points