Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,517 questions
1 answer
Is it possible to encrypt and compress files on self-hosted runtime as part of ADF flow?
I am looking for a way to first encrypt CSV files and then compress before executing the ADF copy flow. The encryption and compression needs to happen on the self-hosted IR. To visualize this a bit Step 1 - Kick-off the ADF flow Step 2 - Encrypt…
asked 2020-11-09T19:06:20.71+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 71%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESW%3C/text%3E%3C/svg%3E)
Sandeep Wadhwa
6
Reputation points
commented 2020-11-23T19:08:26.797+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
1 answer
ADF - Capture the error message when you manually cancel the ADF pipeline run.
I am trying to build an error logging mechanism in Azure Data Factory to capture all the errors into a SQL table. This works fine if a pipeline activity failed itself, I have a stored procedure connecting to the activity (when failed) to write the ADF…
asked 2020-11-11T00:40:48.343+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 73%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EF%3C/text%3E%3C/svg%3E)
Finn
1
Reputation point
commented 2020-11-23T19:02:46.113+00:00
MartinJaffer-MSFT
26,081
Reputation points
2 answers
ADF pipeline remains in Queued status while copying data from As400 or MySQL or Sql Server into BLOB Storage (csv, JSON).
ADF pipeline remains in Queued status while copying data from As400 or MySQL or Sql Server into BLOB Storage (csv, JSON). I have tried both Self Hosted and AZURE integration runtime but it isn't working. My Current location is Toronato , Canada…
asked 2020-11-05T19:19:01.547+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(102.4, 61%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAM%3C/text%3E%3C/svg%3E)
Alinkil, Mithun
21
Reputation points
commented 2020-11-23T08:05:16.603+00:00
PRADEEPCHEEKATLA-MSFT
88,461
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
MongoDB Atlas doesn't appear in my Linked Services list in Azure Data Factory 2
MongoDB Atlas doesn't appear in my Linked Services list in Azure Data Factory 2 I'm trying to create a service link for Atlas mongodb in Microsoft Azure Data Factory v2. According to this link now we can copy data from MongoDB Atlas using Azure Data…
asked 2020-11-09T13:40:50.21+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(38.4, 69%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMS%3C/text%3E%3C/svg%3E)
Mustafa Salaheldin
101
Reputation points
commented 2020-11-23T07:57:12.977+00:00
PRADEEPCHEEKATLA-MSFT
88,461
Reputation points Microsoft Employee
2 answers
Can the limit of maximum 500 event triggers per storage account be increased?
We have a requirement in the project to use approx. 2000 triggeres to check the blob creation events. These blob creation events will have fairly separate files with different names. So no single triggers can be utilized to check the events happening for…
asked 2020-11-17T08:55:42.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 57.99999999999999%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAO%3C/text%3E%3C/svg%3E)
Ankush O Wathodkar
21
Reputation points
answered 2020-11-22T15:35:51.617+00:00
Roman Kiss
2,246
Reputation points
1 answer
One of the answers was accepted by the question author.
Is there a way to use Mongo Atlas like target (sink) in a Azure Data Factory copy o migration operations?
Hi guys, right now we need to use ADF to migrate information from ORACLE (OnPremise using Express Route) to Mongo Atlas (Azure cloud) but there is not sink connector for Mongo Atlas, so we need your help to realize how to do this integration. If…
asked 2020-11-17T15:38:38.507+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(316.8, 92%, 40%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAB%3C/text%3E%3C/svg%3E)
Alejandro Benitez Aragon
41
Reputation points
accepted 2020-11-20T23:42:29.103+00:00
Alejandro Benitez Aragon
41
Reputation points
0 answers
Custom error handling Azure Data Factory
Hi We want to create custom error messages in our pipelines like: File is missing a header record Flow abended: too many incomplete records will be added Too many error records How are people handling these errors in a SQLDB and non-SQL…
asked 2020-11-18T13:48:00.023+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 92%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Poel van der, RE (Ron)
451
Reputation points
commented 2020-11-20T22:44:45.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 44%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESS%3C/text%3E%3C/svg%3E)
Saurabh Sharma
23,796
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Importing data with Lookup fields using Azure Data Factory Common Data Service connector, when there is no ID field in source data
I have a customer who's migrating to a Common Data Service based application and needs a weekly data sync refresh. This data sync will upsert to a table from a CSV in Blob Storage to the Common Data Service for Apps connector. The CSV data provides…
asked 2020-11-05T18:06:04.3+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(57.599999999999994, 71%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERW%3C/text%3E%3C/svg%3E)
Ryan Widmer
21
Reputation points
accepted 2020-11-20T21:05:29.277+00:00
Ryan Widmer
21
Reputation points
1 answer
One of the answers was accepted by the question author.
AZURE DATA FACTORY - Oracle Database
Hi , Is it possible to execute Oracle Database Stored Procedure in Azure Data Factory and output as a CSV File? The Stored Procedure tool seems only available for SQL Server Connection.
asked 2020-06-02T09:42:53.087+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(176, 0%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
KnowledgeSeeker
21
Reputation points
commented 2020-11-20T19:48:03.887+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 41%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECC%3C/text%3E%3C/svg%3E)
Chawla, Charu [PHSA]
1
Reputation point
0 answers
Data Factory copy activity writes a huge file from a tiny file : throttling error ?
Hello, A copy activity has written very large files (over 1TB) when the files should have been tiny (less than 1MB). Here are the details : The activity copy has an SFTP source, binary, and ADLS Gen 2 sink, binary as well. The files have a…
asked 2020-11-12T12:16:26.03+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(99.2, 79%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EEB%3C/text%3E%3C/svg%3E)
Emilien BOUCAUD
21
Reputation points
commented 2020-11-20T19:37:52.27+00:00
Saurabh Sharma
23,796
Reputation points Microsoft Employee
1 answer
Copy Activity writing an empty value from REST API to SQL DW when preview is correct
Hi, I have an copy activity to copy an value from an REST API to an SQL DW. I configured the source and I can even get a correct preview of the value and correct mapping (mapped the name of the value I wanted to the correct sink on the DW by using the…
asked 2020-11-05T13:24:52.36+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(304, 67%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECH%3C/text%3E%3C/svg%3E)
Caio Henrique Segawa Tonetti
1
Reputation point
commented 2020-11-20T08:32:11.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
Can we trigger pipeline on first business day of each month to exclude holidays?
Hi Can we exclude public holidays to trigger Data Factory pipeline ? If not can we achive in Logic apps so that I can call pipeline in a logic app? Also, I want to consider day light saivings to trigger pipeline instead of UTC. Thanks in advance.…
asked 2020-11-12T00:00:38.907+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 49%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKC%3C/text%3E%3C/svg%3E)
Kalyani Chakka
1
Reputation point
commented 2020-11-19T23:34:00.953+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Loading Dynamics CDM with Azure Data Factory (@entityReference)
Using Azure Data Factory to Sink to Dynamics Common Data Model. Source: Reading from MySQL Col1: dpp_authorityid Col2: dpp_acitivitygroup Destination - CDM - dpp_authority Col1: dpp_authorityid Col2: dpp_acitivitygroup (this is Lookup…
asked 2020-11-16T00:25:44.6+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(51.2, 78%, 14%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
VENGALA Sreedhar
101
Reputation points
commented 2020-11-19T20:42:56.36+00:00
MartinJaffer-MSFT
26,081
Reputation points
2 answers
How to deal with long running Azure Functions in Azure Data Factory
I have a Function (HttpTrigger) that can take a bit to run 5-10 minutes. It reads some information from the body of the request to perform some actions: [FunctionName("myFunc")] public static async Task<IActionResult> Run( …
asked 2020-11-16T13:32:28.887+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(307.2, 3%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECH%3C/text%3E%3C/svg%3E)
Christian Hansen
21
Reputation points
commented 2020-11-19T20:41:36.497+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
ADF and data sources of Azure Cloud and On-Prem
All, We need to bring in data into Azure Cloud from external sources that are not in our On-Prem infrastructure. For any data sources inside on-prem we would use Integration Runtime. But considering that it is from a 3rd party vendor which proved is…
asked 2020-11-07T03:23:43.247+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(137.6, 51%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
grajee
351
Reputation points
commented 2020-11-19T17:55:44.307+00:00
Saurabh Sharma
23,796
Reputation points Microsoft Employee
1 answer
Multi Tenant and Event Grid trigger.
Respected, We have our website which supports multiple tenants. We are designing Azure Data Factory where in, each tenant will upload a file in Azure Blob Storage that need to processed and once the file is uploaded then the trigger configured…
asked 2020-11-03T05:43:22.007+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
jigsm
236
Reputation points
commented 2020-11-19T17:53:24.41+00:00
Saurabh Sharma
23,796
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Loading files with ever changing columns
Hi All, I am using ADF to load files from a sftp site into ADLS. The issue is the incoming files are not consistent , column wise. Some times they add columns and there is no way we get notified. Is there a way in ADF where it automatically copies the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 7.000000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
JSON object to array of nested JSON
I have a JSON object in cosmos DB(source) which I need to transform into an array of nested JSON objects(similar to source objects)in Cosmos DB(sink). Using the derived column I am facing an error during typecast of complex json to array. Could I get an…
asked 2020-11-06T08:34:18.233+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 61%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
SARASWAT CHETAN
91
Reputation points
accepted 2020-11-19T04:40:57.81+00:00
SARASWAT CHETAN
91
Reputation points
1 answer
One of the answers was accepted by the question author.
Data factory pipeline shows success when copying csv to sql azure but data does not appear in the table
When running a data factory pipeline to copy a csv file to sql azure, the job shows as a success and confirms that the proper amount of rows were copied but when querying the table in the database, the data does not show.
asked 2020-11-12T20:57:46.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 74%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERA%3C/text%3E%3C/svg%3E)
Robert Anderson
116
Reputation points
accepted 2020-11-17T14:58:58.18+00:00
Robert Anderson
116
Reputation points
2 answers
Azure Data Factory - Paypal - Lookup error
I am trying to execute a lookup to read records from Paypal: This is the query I am running Select Id,Transactions_Index,Sale_Create_Time,Sale_Parent_Payment,Sale_Amount_Details_Subtotal, Sale_Id from Payment_Transactions_Sale Where Sale_Create_Time…
asked 2020-11-16T04:59:23.097+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(3.2, 52%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAD%3C/text%3E%3C/svg%3E)
Ana Del Campo
121
Reputation points Microsoft Employee
answered 2020-11-17T01:22:19.28+00:00
Ana Del Campo
121
Reputation points Microsoft Employee