ADF Issue with partially uploading data to Cosmos DB SQL API

Brahmendra Shukla 0 Reputation points
2024-07-19T15:09:45.43+00:00

I am migrating data from one container to another container in Cosmos DB SQL API, where I only changed the partition key. However, when migrating using ADF, it is not writing the complete data at the sink side. Can you please guide me on how to resolve this issue?

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,599 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,493 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Amira Bedhiafi 22,311 Reputation points
    2024-07-19T16:41:53.22+00:00

    You may need to check the run history and logs of your ADF pipeline for any errors or warnings that could indicate why data is not fully migrating.

    Also don't forget that insufficient throughput on either the source or the sink container can cause incomplete data migration. Try increasing the Request Units (RUs) for both containers during the migration process.

    One other thing, consider using ADF Data Flows instead of Copy Activity for more complex transformations and handling large datasets. Data Flows provide better control over data partitioning and can help in managing the load more effectively.

  2. Oury Ba-MSFT 18,601 Reputation points Microsoft Employee
    2024-07-19T18:43:31.4333333+00:00

    @Brahmendra Shukla Thank you for reaching out.

    First, Is this an offline or online migration?

    If online, would really suggest using Databricks with Spark Connector. It is a little more involved with code and notebooks etc,. Many customers both internal and external have migrated very large volumes for real time/online migration: azure-sdk-for-java/sdk/cosmos/azure-cosmos-spark_3_2-12/Samples/DatabricksLiveContainerMigration at main · Azure/azure-sdk-for-java (github.com)

    If offline, would suggest using native container copy feature in Cosmos DB: https://video2.skills-academy.com/azure/cosmos-db/container-copy

    Change of partition key could be a problem if there are some records that don't contain the field that you have now decided to partition with. If the partition key is changing, you need to ensure that the new partition key is correctly set in the target container. This might involve transforming the data to include the new partition key.

    Regards,

    Oury

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.