Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,029 questions
0 answers
Could someone from Microsoft provide details on SAP Note:-3255746
Hello Team, Could you please provide an insight on ODP context in Synapse Analytics and that ABAP_CDS cannot be use directly through this methodology after SAP Note? If we have to expose SAP Data it has to be through only Odata? is that correct? Have…
asked 2024-07-08T20:38:48.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(73.60000000000001, 22%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EYP%3C/text%3E%3C/svg%3E)
Yash Parikh
0
Reputation points
edited the question 2024-07-08T22:55:44.39+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 4%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
kobulloc-MSFT
25,961
Reputation points Microsoft Employee
1 answer
Is it possible to upgrade the Airflow version of a Workflow Orchestration Manager while it is running?
Currently, only 2.6.3 is available to choose, but when newer versions become available, what is the upgrade path for existing instances? Is it always required to create a completely new instance when upgrading? How would you preserve the Meta-DB with the…
asked 2024-07-03T10:40:56.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 19%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
aeteq
0
Reputation points
edited an answer 2024-07-08T22:05:02.5233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(86.4, 36%, 18%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBM%3C/text%3E%3C/svg%3E)
BhargavaGunnam-MSFT
28,766
Reputation points Microsoft Employee
1 answer
Migration to Azure
Hi friends, we have a requirement to Migrate Denodo to Azure, and I have these questions: What could be the challenges we can anticipate and be aware of? Does the data's size impact the migration's complexity and duration? What tools are the best suited …
asked 2024-07-08T14:32:15.9266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(166.4, 15%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Anshal
2,086
Reputation points
answered 2024-07-08T19:17:42.9066667+00:00
Amira Bedhiafi
18,741
Reputation points
1 answer
primarky key column in target table adf
How to handle data when primary key(1,2,3- auto increament) column has no value in column in the source how to match when column is not available for the same in source
asked 2024-07-06T13:53:34.1133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
285
Reputation points
commented 2024-07-08T17:45:47.52+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
12,105
Reputation points Microsoft Vendor
1 answer
How to flatten nested json array in ADF where the first node does not have key and starts with values which is not constant
I am trying to parse nested JSON in my pipeline to store the values such as table_name, lastSuccessfulWriteTimestamp, totalProcessedRecordsCount, dataFilesPath, schemaHistory_value1, schemaHistory_value2 from below JSON data using ADF dataflow activity.…
asked 2024-07-08T17:22:36.6666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 5%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESH%3C/text%3E%3C/svg%3E)
Satish Hadapad
0
Reputation points
answered 2024-07-08T17:43:52.39+00:00
Azar
21,230
Reputation points MVP
1 answer
ErrorCode=UserErrorInvalidColumnMappingColumnNotFound
I have a staging table and target table with exactly the same schemas, only different data types e.g. nvarchar(max) / nvarchar(50). I'm attempting to insert the data from the staging to the target using a Copy Activity. I have removed parameterised…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 76%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EV%3C/text%3E%3C/svg%3E)
0 answers
Why are so many Workflow Orchestration Manager configurations locked? Especially AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION is annoying.
What is the reason that so many configuration settings are locked, some to non-default values? For example, we would like to turn AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION to True but it is currently locked to False. This greatly hinders our workflow:…
asked 2024-07-03T10:45:02.1733333+00:00
aeteq
0
Reputation points
commented 2024-07-08T15:11:40.3033333+00:00
BhargavaGunnam-MSFT
28,766
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
Creating linked service in azure data factory to connect azure storage account
Hi, I have created azure data factory with out "Managed Virtual Network" . Created azure storage account and disabled 'Public network access' in storage account, Both data factory and storage accounts are in same vnet and same subnet. Later, I…
asked 2024-07-05T12:14:11.53+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(188.79999999999998, 37%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EM%3C/text%3E%3C/svg%3E)
Mani
0
Reputation points
commented 2024-07-08T14:06:23.5633333+00:00
PRADEEPCHEEKATLA-MSFT
84,531
Reputation points Microsoft Employee
2 answers
One of the answers was accepted by the question author.
When to use Tumbling Window type trigger
Hi All, Was going through Microsoft document on Triggers available in ADF. Can anyone explain what is the significance of Tumbling Window Trigger type as we can add recurrence in Schedule trigger type too then in what scenario should we use Tumbling…
asked 2020-12-22T13:03:11.23+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(291.2, 27%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPJ%3C/text%3E%3C/svg%3E)
Priya Jha
871
Reputation points
commented 2024-07-08T14:03:59.4166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(192, 88%, 28%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHE%3C/text%3E%3C/svg%3E)
Hamza Ehsan Khan
0
Reputation points
1 answer
How to add new field to request member to input in Microsoft Azure Group
Hi, I have situation like this: I have a group of users, with owners and members. The owners can actually add new members (by typing new members' emails) to the group, to see different reports. However, for the new members, we only want them to see their…
asked 2024-07-08T12:38:44.5233333+00:00
Daniel Hoang Nguyen
0
Reputation points
answered 2024-07-08T13:34:34.4833333+00:00
Amira Bedhiafi
18,741
Reputation points
1 answer
Error when trying to copy data
I am getting this error even though I have checked the source data and the column is there. { "errorCode": "2200", "message": "Failure happened on 'Sink' side.…
asked 2020-10-01T22:36:05.087+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(201.6, 5%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Christopher Aghassi
16
Reputation points
commented 2024-07-08T13:33:41.9066667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(28.799999999999997, 15%, 12%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ED%3C/text%3E%3C/svg%3E)
DPotts
0
Reputation points
1 answer
Synapse Link Question
Hi, Reference ... https://video2.skills-academy.com/en-us/power-apps/maker/data-platform/azure-synapse-link-pipelines?tabs=synapse-analytics The population of the Data Lake from Dataverse appears successful overall. However, I have increased the incremental…
asked 2024-07-08T10:32:23.4033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(70.4, 92%, 16%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKP%3C/text%3E%3C/svg%3E)
Kieran Patrick Wood
0
Reputation points
answered 2024-07-08T13:27:35.77+00:00
Amira Bedhiafi
18,741
Reputation points
1 answer
How do you access the Airflow CLI/API/DB in Workflow Orchestration Manager?
Working with Airflow, one often requires access to the Airflow CLI/API or even access to the underlying meta database. For example: to import or export connections start backfills manage Variables (using scripts) manage DB items that are…
asked 2024-07-03T10:33:38.4866667+00:00
aeteq
0
Reputation points
commented 2024-07-08T12:41:15.45+00:00
Smaran Thoomu
12,105
Reputation points Microsoft Vendor
0 answers
Can we remove the artificial limitation of ADF Snowflake V2 supporting Multiple Statements
Issue with Snowflake V2 https://video2.skills-academy.com/en-us/azure/data-factory/connector-snowflake?tabs=data-factory#differences-between-snowflake-and-snowflake-legacy How can I get feedback to the engineers that are working on this new connector for…
asked 2024-06-25T06:37:47.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 81%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMN%3C/text%3E%3C/svg%3E)
Macolino, Nick (Health)
1
Reputation point
commented 2024-07-08T12:21:23.9866667+00:00
Smaran Thoomu
12,105
Reputation points Microsoft Vendor
0 answers
How can you import data into the Workflow Orchestration Manager Database?
We have been using Airflow on premise for a long time and would like to migrate to WOM without losing our execution history. Is there a way to import data into the Airflow Meta-DB?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 48%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EH%3C/text%3E%3C/svg%3E)
1 answer
How to create sha256 signature for Oracle Cloud integration Rest API ?
Hi Experts, Infra: Azure Synapse -we are using pipelines / API method to get data and create a Data Warehouse. Source: We have an Oracle Cloud Infrastructure (OCI ) as source , that will output the report to Oracle Cloud Storage. OCI has a REST API…
asked 2024-07-04T03:27:38.67+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(272, 41%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECA%3C/text%3E%3C/svg%3E)
Caniut Alex
0
Reputation points
commented 2024-07-08T12:10:31.9233333+00:00
Smaran Thoomu
12,105
Reputation points Microsoft Vendor
0 answers
Copy CSV files to citrix sharefile path (doman.sharefile.com) folder using ADF
Hello, I am trying to do a POC, which is to build a small pipeline and I am trying to export a SQL table (AZ SQL DB) to a CSV file and need to upload it on citrix sharefile path, site url looks like "https://domain.sharefile.com/home/shared".…
asked 2024-07-04T07:11:36.68+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(217.60000000000002, 1%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ES%3C/text%3E%3C/svg%3E)
Sree
0
Reputation points
commented 2024-07-08T12:01:17.4333333+00:00
Smaran Thoomu
12,105
Reputation points Microsoft Vendor
4 answers
how to fetch data from Azure Active Directory(AD) by using either ADF or databricks
To fetch data from Azure Active Directory (AD) using either Azure Data Factory (ADF) or Azure Databricks, Pleae let me know in detail. thanks
asked 2024-07-04T10:41:44.7933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(156.8, 77%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELN%3C/text%3E%3C/svg%3E)
Lakshmi Narayana Sarma Bhamidipati
30
Reputation points
commented 2024-07-08T11:54:28.53+00:00
Smaran Thoomu
12,105
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
complex aggregation before the reporting
Hi friends, we are considering having Microsoft analysis services before the data is pushed to PowerBI. The reason is to do complex aggregations and roll-ups. I did the analysis and Microsoft analysis services is expensive , I am confused. What should be…
1 answer
One of the answers was accepted by the question author.
Data Factory Pipeline, I want to Publish my trigger that I have created through a Http request, is this at all possible.
Hello, I have created a Data Factory Pipeline with help from the below links using a HTTP Rest API call. https://video2.skills-academy.com/en-us/rest/api/datafactory/pipelines/create-or-update?view=rest-datafactory-2018-06-01&tabs=HTTP#code-try-0 …
asked 2024-04-24T11:28:52.42+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(150.4, 78%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ECM%3C/text%3E%3C/svg%3E)
Clinton Morgan
20
Reputation points
commented 2024-07-08T09:50:55.3733333+00:00
Clinton Morgan
20
Reputation points