Azure Data Factory

How to write a Json file from a csv file

Hello I have got a csv file with an array inside, I need to create a sink data file(json) to be loaded within Cosmos DB, any idea about how to do it? With the csv files i dont have any problem to upload within cosmosDb because I didn't have any…

Error when using Alternate Key in CDS connector

I'm using the Common Data Service for Apps connector in Azure Data Factory to load data into Dynamics 365 I've done this successfully before using the entity key. See this question:…

resolve merge conflicts data factory in dev ops

Hi All, I have a Data Factory V2 which i have connected to Azure DevOps git. I inadvertently edited the master branch then went back to my feature branch to do some more development. When I go to create a pull request to go into master I get a merge…

How do I lookup a description for a value from input source file in Azure Data Factory?

Hi, I'm still very new to Azure Data Factory and thusfar have only used the copy activity using input from blob storage and writing to Azure SQL Database. Now I have a situation where I have in input .csv file with 4 fields from blob storage. 3 of…

Azure Data Factory: Flattening/normalizing a cloumn from CSV file using Azure Data Factory activity

HI, I have pulled a csv file from one of our source using ADF and there is one column called "attributes" which contains multiple fields (in the form of key value pairs). Now I want to expand that column into different fields (columns). Below…

How can I connect Azure Data Factory to SFTP server using ssh authentication

We are slowly moving all our ETL to ADF. We have few SFTP servers and we would like to connect those SFTP servers from ADF. Can someone shed some light on SSH key authentication between ADF and SFTP. Thanks in Advance Nikhil

Processing Azure Data Factory Event Trigger Properties

I have a data factory which triggers based on storage blob event. In the triggered event, I see two properties TriggerTime and EventPayload. As I have need to read the Storage Blob related information I am trying to process the EventPayload in the Data…

Can we use Azure Data Factory to do the following?

Hi, there: We are Microsoft Skype for Business(SFB) and Teams data platform team, currently focus on SFB/Teams Audio/video/application sharing quality back end data collection and analysis. For every SFB and Teams Audio/video/application sharing…

Azure Data Factory: Missing data in the Monitor Dashboard for Data Factory Contributor role

Observed a disparity in the Monitor Dashboard view for the Azure Data Factory instance between: user with Owner role at subscription scope, and, user with Data Factory Contributor role at resource group scope From the account with Owner…

Data factory monitoring activity runs runtime consolidation report

Hello team, I have a master pipeline which has For Each and inside the loop, it has more than 10 activities. How/where to find the time taken of each activity run in consolidation. For ex:- Master For Each Iteration 1 Activity 1 Activity 2 …

Data Factory ForEach performance questions with REST API GET requests

I have a pipeline that has a Lookup activity that is querying a single column of data from an Azure SQL table, fetching the first 1000 rows of data. This input is then being used in a ForEach activity in order to use the values in a string…

Azure Data Factory: Link connection Sharepoint

Hello, i'm trying to link in Data Factory a Sharepoint and i have some problems, when i try to copy data i have this warning: The access token generated failed, status code: BadRequest, error message:…

How to send data from one Salesforce to another one?

I have followed documentation, but still get error: Operation on target Copy data1 failed: Failure happened on 'Sink' side.…

Is there any way to store json Array or json object in SQL as a string into Azure SQL datawarehouse or anything to convert the Array into string(in Azure Data Factory).

Is there any way to store json Array or json object in SQL as a string or anything to convert the Array into string. Currently I am using the Data factory to fetch the Azure API data and try to store the same into Azure SQL data warehouse, but some of…

Azure Data factory V2 pipeline trigger stuck with In-progress status?

Actually, we have one pipeline with 30 different schedules based on different pipeline parameter. Some of schedule executed every 10 min and some of executed once in a day. After setup the pipeline it running fine for one day. But after that suddenly…

azure data factory - java.nio.BufferOverflowException

Hi, I'm using copy activity to copy ORC files from DL Gen 2 to Synapse. Some files are running fine but other files fail with this error: What is the reason. I'm using : AutoResolveIntegrationRuntime "errorCode": "2200", …

Can't process ORC files in Data Factory : ErrorCode=ParquetJavaInvocationException

Hi, Our organisation uses ORC formatted files for our central file storage, in our data factory I am unable to process most of the ORC files, I am only able to process really small files (< 1mb) for all the other ORC files the pipeline fails to run. …

What will be the required value in the additional expression? It returns an error saying "conversion failed when converting date or time from character string.

ADF V2 failed activity

On June17th when an stored procedure activity failed, anything that was written to output (aka Print) in the stored procedure was included in the error in ADF. Since July 1st this no longer is included in the output. Is there a way we can get this…

Data Factory Salesforce objects not the same as Objects seen in Salesforce URL

I have created a LinkedService to connect to the company supplied Salesforce URL using the customised url, the supplied user id, password and have acquired the correct security token. This connection tests successfully. It is also the same parameters…