Azure DMS doing a Full Load each time for a FAILED activity on PostgreSQL

K Sateesh Kumar Reddy 0 Reputation points
2024-01-12T17:28:57.5533333+00:00

Looking to migrate a 1 TB PostgreSQL database from on-prem ubuntu VM to Azure PostgreSQL Flexible Server using Azure DMS. Appears Azure DMS is doing a full load each time the migration activity fails (like when the source database is restarted, network related failures) and rerun.

Since this is an online migration I assumed Azure DMS keeps tracks of WAL and restarts the activity from point of termination however that doesn't seem to be the case. I noticed all of the tables were FULL LOADED from source on rerun.

Questions:

  1. Is this normal behaviour of DMS to do a full load each time an activity fails even after initial full load is complete as that doesnt seem like a valid option for PostgreSQL DB of 1 TB.
  2. Why isnt Azure DMS reading from WAL and restarting from the same point of failure?

Test Case using DVDRENTAL database for reference.

To replicate volume and monitor during RETRY I inserted data to actor table and got it to 2.5 GB in size. Created a DMS Project for PostgreSQL and synced changes from source PostgreSQL database (13.9) on Azure ubuntu VM to PostgreSQL flexi server (14.9) using DMS. Screenshot 2024-01-12 210438

I checked counts on source/target and see the data replicated by DMS as expected. The source database on VM was restarted. The DMS activity which was RUNNING went to a FAILED state. When attempting to restart the activity with RETRY on Azure portal to continue catching up it was observed DMS does a full load again which wouldnt be an ideal case for a 1 TB database. User's image

Azure Database Migration service
Azure Database for PostgreSQL
{count} votes