Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,518 questions
1 answer
Data flow keeps at state "AcquiringCompute", but in activity overview already failed.
Data flow keeps at state "AcquiringCompute", but in activity overview it already failed. The pipline therefore stays "In Progress" endlessly. The error message: "Hit unexpected exception, please retry. If the problem persists,…
asked 2024-08-27T11:59:08.8533333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(265.6, 3%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHR%3C/text%3E%3C/svg%3E)
Hoeks, Ralph
0
Reputation points
commented 2024-08-30T09:31:51.0733333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(268.8, 8%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EST%3C/text%3E%3C/svg%3E)
Smaran Thoomu
14,140
Reputation points Microsoft Vendor
1 answer
One of the answers was accepted by the question author.
Error in SOQL Query with Salesforce V2 dataset
Hi, I'm attempting to use the SOQL Query option to filter columns in the Salesforce dataset, but I'm running into the following error. Any help would be appreciated. Thanks!
asked 2024-08-08T12:42:16.41+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(35.2, 94%, 13%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERS%3C/text%3E%3C/svg%3E)
Roxan Sadia
20
Reputation points
accepted 2024-08-30T08:11:09.9433333+00:00
Roxan Sadia
20
Reputation points
2 answers
Can you give me example for shared responsibility and service model related to like facebook?
I started learning azure i cannot see the big picture how it works.can i get an example like how facebook or instagram works in cloud computing based on their shared responsibility and their service model?
asked 2024-08-14T14:06:36.5166667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(0, 13%, 10%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMA%3C/text%3E%3C/svg%3E)
muhammad asyraf norudin
0
Reputation points
commented 2024-08-30T07:59:17.82+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(198.4, 69%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESR%3C/text%3E%3C/svg%3E)
Sai Raghunadh M
75
Reputation points Microsoft Vendor
1 answer
Input Payload is invalid, validation result - 'The stream was already consumed. It cannot be read again.'
Hello from last night i am receiving the following error on all my pipelines.
asked 2024-08-29T05:49:19.8666667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(121.6, 83%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENZ%3C/text%3E%3C/svg%3E)
Ninoslav Zekic
10
Reputation points
commented 2024-08-30T07:28:36.72+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(211.20000000000002, 4%, 30%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EO%3C/text%3E%3C/svg%3E)
Ola
0
Reputation points
2 answers
How to Implement Microsoft Authenticator in Apps like ASPNET Web
We have application projects and we have idea to implement OTP Microsoft Authenticator in our apps. On the login request there will be authentication using OTP without password. Our Apps using ASP.NET web based. And in our company had implement…
asked 2024-08-29T14:02:00.0033333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 33%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHA%3C/text%3E%3C/svg%3E)
Henokh Arief (PKWT)
0
Reputation points
commented 2024-08-30T06:12:46.6133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(262.40000000000003, 36%, 35%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ELH%3C/text%3E%3C/svg%3E)
Lan Huang-MSFT
29,166
Reputation points Microsoft Vendor
1 answer
ADF-Global parameters inside Linked Service
Hi all! My company is starting to make use of DevOps pipeline to share the same code in all the productive and not productive envirorments. Before this, everything was wild. For example, the linked service for the Azure KeyVault has the url-link written…
asked 2023-10-01T08:53:07.1+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 39%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDA%3C/text%3E%3C/svg%3E)
Dario Antolini
21
Reputation points
commented 2024-08-30T01:40:05.0066667+00:00
Mathias Denzin
0
Reputation points
1 answer
One of the answers was accepted by the question author.
Need Guidance on Deleting Folders with Azure Data Factory in ADLS
I have a pipeline designed to remove certain folders from ADLS. My folder structure is organized as…
asked 2023-11-05T18:40:32.5433333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 0%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPR%3C/text%3E%3C/svg%3E)
Pravalika-randstad
240
Reputation points
commented 2024-08-29T20:16:35.3233333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(297.6, 52%, 38%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EWJ%3C/text%3E%3C/svg%3E)
Whitaker, Josh
5
Reputation points
1 answer
How does Azure Data Factory's Data Copy REST API pagination rules work, is there any parallelisation and can this be disabled to only copy one page at a time?
Hello, I'm using the copy data activity in a Azure Data Factory pipeline to copy data from a REST API data source (the pipeline will soon be migrated to a Synapse one). A recent update to the source API has stumped me as using the built in pagination…
asked 2024-08-29T16:56:09.3266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 54%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Kit MacInnes-Manby
0
Reputation points
commented 2024-08-29T17:08:22.7566667+00:00
Kit MacInnes-Manby
0
Reputation points
2 answers
ADF Lookup Activity not returning values from Snowflake
Hi there, We have a pipeline that uses a lookup activity to run a script against Snowflake, return the first row of data and assign that returned data into a variable. The lookup code is: { "name": "Last Run", …
asked 2024-04-10T16:25:53.1266667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(160, 57.99999999999999%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EML%3C/text%3E%3C/svg%3E)
Mike Langlois
15
Reputation points
commented 2024-08-29T15:51:43.27+00:00
Mike Langlois
15
Reputation points
1 answer
Column truncate issue - Copy file to DB ADF pipeline
Hi All, I have a pipeline where I am copying data from a file to a database. In the file, I have a few columns with extra spaces that I need to trim or truncate. I am not getting any option to use any transformation or function (dynamic content) on the…
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(96, 24%, 19%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
1 answer
One of the answers was accepted by the question author.
How can we use insert-update function in synapse analytics notebook while storing data to blob and database
I am beginner to synapse analytics. Trying save data to blob and DB using below code. But showing "AnalysisException: /mnt/usprodannex1/usprodgold_layer/ is not a Delta table." error. Note imported required library( from delta.tables import *)…
asked 2024-08-28T13:56:09.35+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(112.00000000000001, 35%, 20%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESM%3C/text%3E%3C/svg%3E)
Shrimathi M
40
Reputation points
commented 2024-08-29T13:50:23.55+00:00
Shrimathi M
40
Reputation points
0 answers
How do I use Data Factory with private resources in an Azure Hub and Spoke topology
I'm trying to set up a Data Factory pipeline copy job between an Azure file share hosted on a storage account and azure blob storage. The file share and the blob storage are on separate subscriptions and are connected to two separate VNETs. These VNETs…
asked 2024-08-06T17:07:21.97+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(278.4, 7.000000000000001%, 36%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMB%3C/text%3E%3C/svg%3E)
Miles Barnard
0
Reputation points
edited the question 2024-08-29T12:42:58.23+00:00
KapilAnanth-MSFT
43,741
Reputation points Microsoft Employee
1 answer
Mappings are not working when "Enable staging" is checked in ADF copy activity. is there a solution?
Hello, I'm using a copy data activity in azure data factory (ADF) to move data from a CSV file to SQL server. I enabled the staging in the settings of the copy activity and pointed to a blob storage, the problem is that the columns mapping is not working…
asked 2024-08-27T07:22:08.81+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 13%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMH%3C/text%3E%3C/svg%3E)
Mario Hassrouty
0
Reputation points
commented 2024-08-29T11:52:34.5133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(288, 61%, 37%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EP%3C/text%3E%3C/svg%3E)
phemanth
9,935
Reputation points Microsoft Vendor
2 answers
One of the answers was accepted by the question author.
US government regulations with azure
I am looking for detailed documentation on ( Family Educational Rights and Privacy Act) FERPA and I am looking for useful information. Still, I could not get a detailed implementation of FERPA with various Azure services such as ADLS, Azure Databricks,…
asked 2024-08-27T16:08:32.4133333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(243.2, 88%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
azure_learner
200
Reputation points
commented 2024-08-29T11:14:12.34+00:00
azure_learner
200
Reputation points
2 answers
How to solve issue with salesforce connector
The previous Salesforce legacy connector allowed SQL queries, which was useful for handling data issues in certain objects. The new connector only supports SOQL queries e.g.,select fields(standard) from xyz ,lacking support for custom field selection or…
asked 2024-08-10T06:47:51.46+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(233.6, 24%, 32%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMP%3C/text%3E%3C/svg%3E)
Mayur Patil
5
Reputation points
commented 2024-08-29T09:52:47.37+00:00
Smaran Thoomu
14,140
Reputation points Microsoft Vendor
1 answer
Synapse Notebook running manually but not throught the pipeline in Synapse (with access policies defined in Key vaults for both my account and the Synapse Workspace))
creds = mssparkutils.credentials.getSecret('fmd-prod-db-readonly','Analytics-JSON') #Define credentials from json credentials = Credentials.from_service_account_info(json.loads(creds)) client = BetaAnalyticsDataClient(credentials=credentials) Im using…
asked 2023-12-29T08:24:08.0466667+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(76.8, 67%, 17%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAS%3C/text%3E%3C/svg%3E)
Abhinandan Shrestha
20
Reputation points
commented 2024-08-29T09:47:53.7333333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(249.60000000000002, 3%, 34%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EJZ%3C/text%3E%3C/svg%3E)
Jesin Zakaria
0
Reputation points Microsoft Employee
1 answer
Validation
Hi, 3+2 ; select case when count(emp) = coun(dept) then pass or fall end from emp (How to validate condition and send extract to blob storage as results.. Pls share screenshot
asked 2024-08-25T15:01:42.18+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(204.8, 13%, 29%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EVS%3C/text%3E%3C/svg%3E)
Vineet S
730
Reputation points
commented 2024-08-29T08:06:36.4133333+00:00
AnnuKumari-MSFT
32,661
Reputation points Microsoft Employee
1 answer
One of the answers was accepted by the question author.
How to troubleshoot Hit unexpected exception in Azure Data Factory.
Hello, Today I've been getting the following errors in my Data Factory runs: Hit unexpected exception, please retry. If the problem persists, please contact Azure Support" According to this thread:…
asked 2024-08-27T07:52:20.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(172.8, 88%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESD%3C/text%3E%3C/svg%3E)
Simon De Meester
20
Reputation points
commented 2024-08-29T06:44:12.6433333+00:00
Simon De Meester
20
Reputation points
2 answers
Getting error with ADF dataverse connector
Hi, I am using a copy activity to populate 'ownerid' column of dataverse Account entity. I got the ownerid by a lookup from dataverse 'systemuser; entity in a previous step. But I am getting the below error. Please…
asked 2024-08-22T12:30:55.73+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(118.4, 16%, 21%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EPS%3C/text%3E%3C/svg%3E)
Pankaj Singh
20
Reputation points
commented 2024-08-29T06:41:29.4366667+00:00
AnnuKumari-MSFT
32,661
Reputation points Microsoft Employee
1 answer
Self Hosted Integration Runtime Connection Failed with Azure Cosmos DB for PostgreSQL
In datafactory we have a step which connects with Azure CosmosDB for PostgreSQL to copy the some data to other DB using self hosted integration runtime, Its was working fine till last week. Suddenly there is intermittent connectivity issue & we are…
asked 2024-08-28T06:35:25.3933333+00:00
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 3%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EFA%3C/text%3E%3C/svg%3E)
Farhan Abdul Shakoor
0
Reputation points
edited a comment 2024-08-29T06:16:28.4466667+00:00
Farhan Abdul Shakoor
0
Reputation points