Azure Data Factory Usage

Question

We have some use cases where we are connecting to a customer database and fetching the data onto our database using Azure Data Factory. Once we copy this data into our database into an isolated table we would like to process this with some business logic and copy them into multiple tables with relationship mapping eg. we need to read each row and create data into the first table with some business logic and then process the same record for some more columns and relate to the first data we created in a different table and also encrypt the data in the table.
Another query is if the above approach works is there a way we can create a pipeline for this entire task and expose this via an API to our application to process some excel data which processes a batch of documents.
Appreciate if we can get some help on what can be achieved and what cannot. Any other possible solutions with azure which can address these are also welcome.

Accepted Answer

Hi @Koteswara Pentakota ,

Thank you for posting query in Microsoft Q&A Platform.

As per my understanding, you created a ADF pipeline that fetches data from customer databases and load it to your database table. now you would like to fetch data from your database table and apply some business logic on top of that data and then load it to different tables. Please correct me if I am wrong.

Here, what you mean when you say apply some business logic on data? If it means applying some transformations on your data, then yes can do that using mapping data flows. Mapping data flows allows us to transform our source data according to need and load it to sink systems.

Click here to know more about mapping data flow. This video also explains about mapping data flows.

When you say apply some business logic, if that means executing some customer code (.Net or python etc.) on your data, then Azure functions are solution to use here.

Azure Functions is a serverless solution that allows you to write less code, maintain less infrastructure, and save on costs. Instead of worrying about deploying and maintaining servers, the cloud infrastructure provides all the up-to-date resources needed to keep your applications running.

Click here to know more about azure functions. This video also explains about Azure functions on high level.

You can make use of pre-copy SQL script option in copy activity or pre-copy or post-copy script options in Sink transformation to execute our custom SQL. So, if you are trying to generate relationship between your tables which are not exists then you can consider using them. Or there is script activity also which helps to run script directly. So, you can leverage script activity too. Click here to know about script activity. This video also explains script activity.

If you are looking for ADF to automatically guess relations based on data and create relations automictically directly, then that is not possible.

Azure data factory cannot encrypt data out of box directly. We should either consider using Custom activity or Azure Function activity. Please check below Q&A thread where similar discussion is available.
https://video2.skills-academy.com/en-us/answers/questions/280549/how-to-configure-the-adf-pipe-line-to-encrypt-and.html

Regarding another ask of exposing ADF pipeline as API, Azure set of REST APIs created to work with Azure resources to automate tasks or etc. So, we can make a API call to ADF pipeline line to trigger it automatically based on our needs. Below link has the documentation for REST APIs related to ADF.
https://video2.skills-academy.com/en-us/rest/api/datafactory/pipelines

Hope this helps. Please let us know if any further queries. If something is missing in my understanding of ask, then kindly share more details on ask by breaking them in to simple tasks with sample examples, so that We can help better. Thank you.

-----------------

Please consider hitting Accept Answer button. Accepted answers help community as well.

Share via

Azure Data Factory Usage

0 additional answers