Azure data factory reporting

CORONADO GRANADOS Diana Milena 126 Reputation points
2020-07-17T12:04:25.027+00:00

Hello

I'm using ADF in order to develop an ETL process, at the end of the process I need to have a txt file with these kind of numbesrs:
number of files processed
Number of records processed (per file)
Number of records not processed (per file)

there is a way to do it?? maybe creating an activity to read number of records?? If someone know or deal with a similar problem could help me?

many thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,566 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Vaibhav Chaudhari 38,721 Reputation points
    2020-07-17T12:19:22.797+00:00

    You could make use of Stored Procedure activity. Run SPs after the activities to log the data to table. Refer below blog to implement something similar. It has to be customized based your requirement

    https://www.mssqltips.com/sqlservertip/6320/logging-azure-data-factory-pipeline-audit-data/

    If the response helped, do "Accept Answer" and upvote it -- Vaibhav

    0 comments
  2. HimanshuSinha-msft 19,471 Reputation points Microsoft Employee
    2020-09-01T20:14:17.857+00:00

    Hello @CORONADO GRANADOS Diana Milena ,

    We can always capture the number of the records inserted on the sink side by using the @{activity('Copy-Table').output.rowsCopied}' , but in case if want to capure how many records were NOT processed , we can use take the difference between of @{activity('Copy-Table').output.rowsRead} and the rows copied( expression shared above ) .

    Hope this helps .

    Thanks
    Himanshu

    0 comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.