Copy last data from two PostgreSQL Db by Azure Data Factory

Mohsen Akhavan 746 Reputation points
2021-05-25T09:08:56.637+00:00

I have a Postgresql database on VM and also, I have an Azure Database for PostgreSQL. I want to know is there any way to use the Azure Data factory for copying the last data?
For example, Daily run a pipeline and copy the last data then added to a table from VM to Azure.
I test copy data, but every time the task run, the pipeline copy all table data and add it to the destination table.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,024 questions
Azure Database for PostgreSQL
Accepted answer
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-05-25T19:30:35.34+00:00

    Hello @Mohsen Akhavan ,
    Thanks for the ask and using the Microsoft Q&A platform .
    If I understand the ask , I think you want to only move the delta data and not all the data . if thats the case you will have to use the watermark logic .
    Please do read about the same here .

    Please do let me know how it goes .
    Thanks
    Himanshu
    Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    1 person found this answer helpful.
    0 comments

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest