Azure Data Factory key column watermakr workflow?

Anmol Ganju 1 Reputation point
2020-10-29T16:20:43.833+00:00
  1. How to Incrementally copy data using Identity column or incrementing key column instead of time-stamp. We checked many blogs but didn't find how to implement this.
  2. This mechanisms is going to be used for ‘N’ # of tables. How can we implement “For-each loop” of multiple table in this case?
  3. How can we implement, Exception handling and logging in ADF pipeline:
    • Way to recover and retry attempt
    • Abort the operation and some action to occur to store those record separately
  4. If a table throws an error, instead of terminating the pipeline ,the error part should be stored in logs and version of that table shouldn't get updated.
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,586 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,471 Reputation points Microsoft Employee
    2020-10-30T19:56:40.41+00:00

    Hello @Anmol Ganju ,

    Thanks for the ask and using the forum .

    Since you mentioned Identity column I am assuming that your source is a DB and let me assume that the sink is either a DB or a blob . If its a DB , please read tables(s) ( before you start the copy activity ) and get the max identity column value , you can use this value to create a dynamic query with the "Where clause" in the copy activity and only get the latest records . if you have sink as blob , you will have a create a watermark table and update the table when you write the records to the blob .

    In the copy activity you do have a option of fault tolerance , i am not sure if that gone through that . On the ask on exception handling , you can use the retry option . For logging errors may be you can see if you can see something like this .

    Let me know if you have any further queries .

    Thanks
    Himanshu

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.