Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,568 questions
0 answers
Data Factory copy activity writes a huge file from a tiny file : throttling error ?
Hello, A copy activity has written very large files (over 1TB) when the files should have been tiny (less than 1MB). Here are the details : The activity copy has an SFTP source, binary, and ADLS Gen 2 sink, binary as well. The files have a…
asked 2020-11-12T12:16:26.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 79%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEB%3C/text%3E%3C/svg%3E)
Emilien BOUCAUD
21
Reputation points
commented 2020-11-20T19:37:52.27+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,801
Reputation points Microsoft Employee
1 answer
Copy Activity writing an empty value from REST API to SQL DW when preview is correct
Hi, I have an copy activity to copy an value from an REST API to an SQL DW. I configured the source and I can even get a correct preview of the value and correct mapping (mapped the name of the value I wanted to the correct sink on the DW by using the…
asked 2020-11-05T13:24:52.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 67%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECH%3C/text%3E%3C/svg%3E)
Caio Henrique Segawa Tonetti
1
Reputation point
commented 2020-11-20T08:32:11.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Can we trigger pipeline on first business day of each month to exclude holidays?
Hi Can we exclude public holidays to trigger Data Factory pipeline ? If not can we achive in Logic apps so that I can call pipeline in a logic app? Also, I want to consider day light saivings to trigger pipeline instead of UTC. Thanks in advance.…
asked 2020-11-12T00:00:38.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 49%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKC%3C/text%3E%3C/svg%3E)
Kalyani Chakka
1
Reputation point
commented 2020-11-19T23:34:00.953+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Loading Dynamics CDM with Azure Data Factory (@entityReference)
Using Azure Data Factory to Sink to Dynamics Common Data Model. Source: Reading from MySQL Col1: dpp_authorityid Col2: dpp_acitivitygroup Destination - CDM - dpp_authority Col1: dpp_authorityid Col2: dpp_acitivitygroup (this is Lookup…
asked 2020-11-16T00:25:44.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 78%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
VENGALA Sreedhar
101
Reputation points
commented 2020-11-19T20:42:56.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
2 answers
How to deal with long running Azure Functions in Azure Data Factory
I have a Function (HttpTrigger) that can take a bit to run 5-10 minutes. It reads some information from the body of the request to perform some actions: [FunctionName("myFunc")] public static async Task<IActionResult> Run( …
asked 2020-11-16T13:32:28.887+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 3%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECH%3C/text%3E%3C/svg%3E)
Christian Hansen
21
Reputation points
commented 2020-11-19T20:41:36.497+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
ADF and data sources of Azure Cloud and On-Prem
All, We need to bring in data into Azure Cloud from external sources that are not in our On-Prem infrastructure. For any data sources inside on-prem we would use Integration Runtime. But considering that it is from a 3rd party vendor which proved is…
asked 2020-11-07T03:23:43.247+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 51%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
grajee
351
Reputation points
commented 2020-11-19T17:55:44.307+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
1 answer
Multi Tenant and Event Grid trigger.
Respected, We have our website which supports multiple tenants. We are designing Azure Data Factory where in, each tenant will upload a file in Azure Blob Storage that need to processed and once the file is uploaded then the trigger configured…
asked 2020-11-03T05:43:22.007+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jigsm
236
Reputation points
commented 2020-11-19T17:53:24.41+00:00
Saurabh Sharma
23,801
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Loading files with ever changing columns
Hi All, I am using ADF to load files from a sftp site into ADLS. The issue is the incoming files are not consistent , column wise. Some times they add columns and there is no way we get notified. Is there a way in ADF where it automatically copies the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 7.000000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
JSON object to array of nested JSON
I have a JSON object in cosmos DB(source) which I need to transform into an array of nested JSON objects(similar to source objects)in Cosmos DB(sink). Using the derived column I am facing an error during typecast of complex json to array. Could I get an…
asked 2020-11-06T08:34:18.233+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 61%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
SARASWAT CHETAN
91
Reputation points
accepted 2020-11-19T04:40:57.81+00:00
SARASWAT CHETAN
91
Reputation points
1 answer
One of the answers was accepted by the question author.
Data factory pipeline shows success when copying csv to sql azure but data does not appear in the table
When running a data factory pipeline to copy a csv file to sql azure, the job shows as a success and confirms that the proper amount of rows were copied but when querying the table in the database, the data does not show.
asked 2020-11-12T20:57:46.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 74%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Robert Anderson
116
Reputation points
accepted 2020-11-17T14:58:58.18+00:00
Robert Anderson
116
Reputation points
2 answers
Azure Data Factory - Paypal - Lookup error
I am trying to execute a lookup to read records from Paypal: This is the query I am running Select Id,Transactions_Index,Sale_Create_Time,Sale_Parent_Payment,Sale_Amount_Details_Subtotal, Sale_Id from Payment_Transactions_Sale Where Sale_Create_Time…
asked 2020-11-16T04:59:23.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 52%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Ana Del Campo
121
Reputation points Microsoft Employee
answered 2020-11-17T01:22:19.28+00:00
Ana Del Campo
121
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
Azure data factory - query pipeline by data annotations
Is there a way to programmatically query (using .net SDK) list of data factory pipelines by data annotations? I can set data annotation when I run pipeline as explained here, but not sure how to query pipelines or filter pipelines using the same?…
asked 2020-10-23T21:37:28.577+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(25.6, 84%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJV%3C/text%3E%3C/svg%3E)
JAY VORA
21
Reputation points
commented 2020-11-16T23:38:22.617+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
How we can get uptime for all the resources in azure.
Hi guys, I have to check the uptime for the resources in Azure. As for some resources, we can check directly from the Azure portal. I'm able to see uptime for the storage account and SQL database but how can I get the uptime for other resources such…
asked 2020-11-07T07:03:54.573+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 21%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAG%3C/text%3E%3C/svg%3E)
Ashima Gupta
66
Reputation points
commented 2020-11-16T22:45:46.163+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
0 answers
Expiry Datetime incorrect for Tumbling Window Trigger Catchup Run?
I have a pipeline that primarily has 2 copy activities . The sink for the copy activities is ADLS Gen 1, so it also sets an expiry datetime for each of the files copied over. It uses a tumbling window trigger with a self-dependency. Recently, the…
asked 2020-11-06T19:26:33.123+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 36%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Rasika Athavale
1
Reputation point Microsoft Employee
commented 2020-11-16T22:40:37.95+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
Logs are missing in the Log Analytics Workspace - Azure Data Factory starting 10/23/2020
Hello, Our ADF and the log analytic workspace are hosted under West US2, same as our ADF, and since 10/23, we’ve noticed several pipeline runs do not come with the Succeeded or Failed log, but just InProgress or Queued. It threw off our query since…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 98%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHH%3C/text%3E%3C/svg%3E)
2 answers
Data Factory Loading multiple csv files. Is there any way of dealing with column header differences?
I have csvs in a datalake. Im coping them into SQL using the copy activity the File Path type is a wildcard file path Very occasionally there are slight differences between the column headers. Is there any way that this can be dealt with rather…
asked 2020-11-05T12:56:42.187+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 27%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDE%3C/text%3E%3C/svg%3E)
Debbie Edwards
521
Reputation points
commented 2020-11-16T22:16:54.58+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Getting TlsVersionNotPermitted error when SSIS package executing from ADF
Hi, I have created one SSIS package which is reading data from Storage account and putting that into Azure DB. This is working perfectly well when executing from Visual Studio. But when I am triggering it after deployment in ADF, getting below error. …
asked 2020-11-12T18:01:00.723+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(246.4, 30%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
Vikas
21
Reputation points
commented 2020-11-16T18:10:09.6+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
One of the answers was accepted by the question author.
Query Azure Function HTTP Get from Azure Data Factory
I have a function like: public static async Task<IActionResult> Run( [HttpTrigger(AuthorizationLevel.Function, "get", Route = null)] HttpRequest req, ILogger log) { string dateFromString =…
asked 2020-11-13T12:22:03.513+00:00
Christian Hansen
21
Reputation points
accepted 2020-11-16T13:33:21.307+00:00
Christian Hansen
21
Reputation points
2 answers
Azure data factory web activity body
How do I use parameters in the body of Push method in data factory? It seems to allow static values or a parameter but not both. Say I want my message body as {"count" :@Anonymous } But this isn't working, any ideas
asked 2020-11-10T04:14:56.33+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 7.000000000000001%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Accelerator
1
Reputation point
commented 2020-11-16T06:31:20.42+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Is it possible to rename file name with MD5 hash encoding in Azure data factory?
I have this scenario where I have a blob with multiple binary files and I would like to copy these files into another blob storage using Azure data factory. The issue is my destination file name should be MD5 encoded . Now I would like to know if it…
asked 2020-10-05T12:46:22.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 85%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
ankit kumar
101
Reputation points
commented 2020-11-14T00:03:32.147+00:00
MartinJaffer-MSFT
26,081
Reputation points