Is it possible to encrypt and compress files on self-hosted runtime as part of ADF flow?
I am looking for a way to first encrypt CSV files and then compress before executing the ADF copy flow. The encryption and compression needs to happen on the self-hosted IR. To visualize this a bit Step 1 - Kick-off the ADF flow Step 2 - Encrypt…
ADF - Capture the error message when you manually cancel the ADF pipeline run.
I am trying to build an error logging mechanism in Azure Data Factory to capture all the errors into a SQL table. This works fine if a pipeline activity failed itself, I have a stored procedure connecting to the activity (when failed) to write the ADF…
ADF pipeline remains in Queued status while copying data from As400 or MySQL or Sql Server into BLOB Storage (csv, JSON).
ADF pipeline remains in Queued status while copying data from As400 or MySQL or Sql Server into BLOB Storage (csv, JSON). I have tried both Self Hosted and AZURE integration runtime but it isn't working. My Current location is Toronato , Canada…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
MongoDB Atlas doesn't appear in my Linked Services list in Azure Data Factory 2
MongoDB Atlas doesn't appear in my Linked Services list in Azure Data Factory 2 I'm trying to create a service link for Atlas mongodb in Microsoft Azure Data Factory v2. According to this link now we can copy data from MongoDB Atlas using Azure Data…
![](https://techprofile.blob.core.windows.net/images/VfQFAmOikEWfBHko2XlWTA.png?8D7F33)
Can the limit of maximum 500 event triggers per storage account be increased?
We have a requirement in the project to use approx. 2000 triggeres to check the blob creation events. These blob creation events will have fairly separate files with different names. So no single triggers can be utilized to check the events happening for…
![](https://techprofile.blob.core.windows.net/images/5ylEgP8_AwAAAAAAAAAAAA.png?8D80EE)
Is there a way to use Mongo Atlas like target (sink) in a Azure Data Factory copy o migration operations?
Hi guys, right now we need to use ADF to migrate information from ORACLE (OnPremise using Express Route) to Mongo Atlas (Azure cloud) but there is not sink connector for Mongo Atlas, so we need your help to realize how to do this integration. If…
Custom error handling Azure Data Factory
Hi We want to create custom error messages in our pipelines like: File is missing a header record Flow abended: too many incomplete records will be added Too many error records How are people handling these errors in a SQLDB and non-SQL…
Importing data with Lookup fields using Azure Data Factory Common Data Service connector, when there is no ID field in source data
I have a customer who's migrating to a Common Data Service based application and needs a weekly data sync refresh. This data sync will upsert to a table from a CSV in Blob Storage to the Common Data Service for Apps connector. The CSV data provides…
AZURE DATA FACTORY - Oracle Database
Hi , Is it possible to execute Oracle Database Stored Procedure in Azure Data Factory and output as a CSV File? The Stored Procedure tool seems only available for SQL Server Connection.
Data Factory copy activity writes a huge file from a tiny file : throttling error ?
Hello, A copy activity has written very large files (over 1TB) when the files should have been tiny (less than 1MB). Here are the details : The activity copy has an SFTP source, binary, and ADLS Gen 2 sink, binary as well. The files have a…
Copy Activity writing an empty value from REST API to SQL DW when preview is correct
Hi, I have an copy activity to copy an value from an REST API to an SQL DW. I configured the source and I can even get a correct preview of the value and correct mapping (mapped the name of the value I wanted to the correct sink on the DW by using the…
Can we trigger pipeline on first business day of each month to exclude holidays?
Hi Can we exclude public holidays to trigger Data Factory pipeline ? If not can we achive in Logic apps so that I can call pipeline in a logic app? Also, I want to consider day light saivings to trigger pipeline instead of UTC. Thanks in advance.…
Loading Dynamics CDM with Azure Data Factory (@entityReference)
Using Azure Data Factory to Sink to Dynamics Common Data Model. Source: Reading from MySQL Col1: dpp_authorityid Col2: dpp_acitivitygroup Destination - CDM - dpp_authority Col1: dpp_authorityid Col2: dpp_acitivitygroup (this is Lookup…
How to deal with long running Azure Functions in Azure Data Factory
I have a Function (HttpTrigger) that can take a bit to run 5-10 minutes. It reads some information from the body of the request to perform some actions: [FunctionName("myFunc")] public static async Task<IActionResult> Run( …
ADF and data sources of Azure Cloud and On-Prem
All, We need to bring in data into Azure Cloud from external sources that are not in our On-Prem infrastructure. For any data sources inside on-prem we would use Integration Runtime. But considering that it is from a 3rd party vendor which proved is…
Multi Tenant and Event Grid trigger.
Respected, We have our website which supports multiple tenants. We are designing Azure Data Factory where in, each tenant will upload a file in Azure Blob Storage that need to processed and once the file is uploaded then the trigger configured…
Loading files with ever changing columns
Hi All, I am using ADF to load files from a sftp site into ADLS. The issue is the incoming files are not consistent , column wise. Some times they add columns and there is no way we get notified. Is there a way in ADF where it automatically copies the…
JSON object to array of nested JSON
I have a JSON object in cosmos DB(source) which I need to transform into an array of nested JSON objects(similar to source objects)in Cosmos DB(sink). Using the derived column I am facing an error during typecast of complex json to array. Could I get an…
Data factory pipeline shows success when copying csv to sql azure but data does not appear in the table
When running a data factory pipeline to copy a csv file to sql azure, the job shows as a success and confirms that the proper amount of rows were copied but when querying the table in the database, the data does not show.
Azure Data Factory - Paypal - Lookup error
I am trying to execute a lookup to read records from Paypal: This is the query I am running Select Id,Transactions_Index,Sale_Create_Time,Sale_Parent_Payment,Sale_Amount_Details_Subtotal, Sale_Id from Payment_Transactions_Sale Where Sale_Create_Time…