Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,517 questions
1 answer
One of the answers was accepted by the question author.
APIs to monitor pipleine?
Hi, Are there any APIs to monitory Pipeline, Dataflow, Activities in Azure Data Factory? The API can directly be used in .Net application.
asked 2020-10-26T12:21:49.69+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 61%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAV%3C/text%3E%3C/svg%3E)
Abhishek Vyas
101
Reputation points
commented 2020-10-29T17:02:10.54+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 70%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMM%3C/text%3E%3C/svg%3E)
MartinJaffer-MSFT
26,081
Reputation points
1 answer
One of the answers was accepted by the question author.
How can I check if my parameter consists of one or more attributes?
Hi see attachment When the string of the incoming pipeline parameter contains one attribute I should use byName, otherwise byNames. That that would presume I should build two different flows, which for the rest do the same. How can…
asked 2020-10-26T11:52:26.533+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 92%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPV%3C/text%3E%3C/svg%3E)
Poel van der, RE (Ron)
451
Reputation points
commented 2020-10-29T16:59:41.217+00:00
MartinJaffer-MSFT
26,081
Reputation points
2 answers
One of the answers was accepted by the question author.
Get Dynamics 365 OptionSet labels in Copy Activity
Is there any way to get optionset labels from a Dynamics 365 entity using a copy activity? All I can seem to get is the option number when I query an entity (using Common data services and Dynamics 365 linked services) Thanks!
asked 2020-06-17T00:49:24.883+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(220.8, 45%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJA%3C/text%3E%3C/svg%3E)
John Aherne
516
Reputation points
commented 2020-10-29T15:14:41.607+00:00
John Aherne
516
Reputation points
1 answer
One of the answers was accepted by the question author.
Sinking to a Power BI created Data Flow using Data Factory CDM sink
Hi, I've created a Power BI Data Flow with a single entity and I would like to use Data Factory to push data to it. I am unsure what options I should be using in the Data Factory inline CDM sink. I can see I have a model.json file in the Power BI…
asked 2020-10-27T19:29:34.12+00:00
Serverless SQL
216
Reputation points MVP
commented 2020-10-28T19:43:21.393+00:00
Serverless SQL
216
Reputation points MVP
1 answer
ADF "Index out of range" when reading XLSX with hidden columns
@Anonymous @Anonymous @Anonymous Hello everyone, I have a XLSX file generated by a system, and this file has 3 hidden columns on the start (columns A, B, and C). These columns aren't important and can be ignored if necessary. So, I've…
asked 2020-10-20T14:35:23.897+00:00
Alexandre Von Mühlen
1
Reputation point
answered 2020-10-28T18:47:47.067+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 26%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
How to get the latest Json file from any Cloud source to the ADLS whenever the ADF Pipeline runs
Hi Team Whenever we run a pipeline in ADF, how we are getting the latest file from a list of files from a source and copying it into the ADLS Regards Kks
asked 2020-10-06T10:30:30.973+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(147.20000000000002, 10%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EK%3C/text%3E%3C/svg%3E)
kks8589
101
Reputation points
commented 2020-10-28T18:07:12.86+00:00
HimanshuSinha-msft
19,471
Reputation points Microsoft Employee
1 answer
Check Version of Azure Gateway services and upate with latest version
Hi All, Where can i find my Azure gateway service information on the Azure Portal. How can i check what version it is running on and how can i find the latest version and update it? Thanks
0 answers
To get all CosmosDB Mongo API collection using foreach and copy Activity
I have 10 collection in CosmosDB with Mongo API and load in BLOB storage , I can able to do for 1 collection using copy activity but cant do for all collection as I have used for each activity but not able to get all collection name in for each…
asked 2020-10-20T06:58:40.28+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 97%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EG%3C/text%3E%3C/svg%3E)
Girish13
1
Reputation point
commented 2020-10-28T15:29:17.497+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 9%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHM%3C/text%3E%3C/svg%3E)
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
One of the answers was accepted by the question author.
Range and Decimal Precision in Excel to CSV conversion in ADFv2
Hi All, I was exploring Excel as a Source in ADFv2 by trying to convert an excel file into CSV by using copy activity. I faced the below challenges: 1) There is a provision to provide Range of Excel file. How can we provide dynamic range. My excel files…
asked 2020-08-28T13:13:13.297+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 27%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Priya Jha
871
Reputation points
commented 2020-10-28T11:55:28.49+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(89.60000000000001, 84%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELM%3C/text%3E%3C/svg%3E)
Lakshmi Mounika Batchu
6
Reputation points
4 answers
One of the answers was accepted by the question author.
Pause and resume Azure Data Factory pipeline
Respected, I have a requirement where the Azure Data Factory Pipeline will insert data into a staging table, over here I want the pipeline to pause, we will be showing a UI page to user so that they can review the data that is being processed. Once…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(16, 25%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJ%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How to check if my integration run time is upto date
Hi All, Can someone please guide me. I want to know if my integration run time is upto date and if not how to update it. Thanks
asked 2020-10-27T12:41:55.14+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 7.000000000000001%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EB%3C/text%3E%3C/svg%3E)
bk
466
Reputation points
commented 2020-10-27T21:19:06.833+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
Azure data factory - column value not flowing through to sink
I have a simple data factory. Source dataset, alter row (insert or update), sink to destination table. Source and destination are Azure SQLServer I am having trouble with one column. It is defined as tinyint in both databases. When I data…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(300.8, 8%, 39%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EC%3C/text%3E%3C/svg%3E)
0 answers
ADF Schema validation for JSON
Hi Community, I am implementing a feature which is validating a dataset schema, I know that in ADF there is a checkbox in the Mapping Dataflow section which does it automatically if columns or type of the columns do not match against the reference. …
asked 2020-10-20T07:39:12.727+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(6.4, 39%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDB%3C/text%3E%3C/svg%3E)
Daniel Berkes
1
Reputation point
commented 2020-10-27T07:49:50.487+00:00
HarithaMaddi-MSFT
10,136
Reputation points
1 answer
ADF - How to identify the number of partition needed, while using SAP source?
Hi Team, My source is SAP and my table is having billions of records, i'm using CopyData activity with Partition option as "On Int" to retrieve the data faster. I'm passing "20000" as "Max partitions number", the problem…
asked 2020-10-14T11:28:03.437+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 98%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESK%3C/text%3E%3C/svg%3E)
Santhosh Kumar
6
Reputation points
answered 2020-10-26T22:59:45.3+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
One of the answers was accepted by the question author.
Performance Comparison: CDS API vs. Power Automate vs. Data Factory
I'm trying to decide on which solution would be the most efficient for transferring data to a CDS database. I've found performance benchmarks for Azure Data Factory (link here) and was wondering if there was similar data available for using CDS's native…
asked 2020-10-21T18:06:33.347+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 18%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESC%3C/text%3E%3C/svg%3E)
Sam Crawford
21
Reputation points
accepted 2020-10-23T18:52:36.56+00:00
Sam Crawford
21
Reputation points
2 answers
Not able to copy entire files and folders recursively in specific folder in Azure Data Factory
Hi, I want to copy data between two adls gen2 accounts. I am unable to copy all files and subfolders in specific folder to the destination. Please check the attached screenshot which shows directory structure of my source adls account and…
asked 2020-09-23T17:00:15.65+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 28.000000000000004%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Rohit Boddu
466
Reputation points
commented 2020-10-23T17:57:42.163+00:00
MartinJaffer-MSFT
26,081
Reputation points
1 answer
Load data from a website using ADL Data pipelines
Hi All I want to load data from an excel link on a webiste. Is there a good document to show how to do it? I want to load data using ADF. Thanks
asked 2020-10-19T15:09:27.467+00:00
bk
466
Reputation points
commented 2020-10-23T17:56:13.81+00:00
MartinJaffer-MSFT
26,081
Reputation points
2 answers
One of the answers was accepted by the question author.
How to set up automatice trigger for azure file share
Hi All, I want to set up trigger for Azure file share. Requirement is when file is placed in azure file path it should automatically trigger the ADF pipeline to send the file to destination. Kindly provide your inputs on the above query. Thanks in…
asked 2020-10-01T15:52:25.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(108.80000000000001, 21%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Ravikumar Sivasamy
21
Reputation points
answered 2020-10-23T16:07:03.407+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 51%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Ravikumar Sivasamy
1
Reputation point
2 answers
One of the answers was accepted by the question author.
pipeline status shows succeeded if the last activity failed.
my pipeline contains below activity if my last Stored procedure failed (marked), my pipeline status show succeeded instead failed. I added another dummy activity after that last stored procedure (after the success), then it says failed. …
asked 2020-10-20T14:57:20.937+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 91%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERB%3C/text%3E%3C/svg%3E)
Ramalingam, Balamurugan
21
Reputation points
accepted 2020-10-23T11:14:57.713+00:00
Ramalingam, Balamurugan
21
Reputation points
1 answer
One of the answers was accepted by the question author.
Unable to connect to GCS from ADF
Hi Team When creating a linked service in ADF to connect to GCS, the connection is getting failed. we were provided with the GCS service account access id and key and URI Any other permissions need to be taken care at gcs side to establish the…