Failing to copy large dataset from Salesforce using new salesforce V2 connector in Azure data factory

LY 0

I am trying to migrate the salesforce legacy linked service to new salesforce V2 linked service in Azure data factory. The test connection is successful and I am able to copy the smaller datasets using new salesforce V2 linked service. But when i am trying to copy the larger datasets(Around 50GB in my scenario) from salesforce using the new linked service connector, the copy activity is failing with the following exception.

Failure happened on 'Source' side. ErrorCode=SalesforceAPITaskCancelException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Getting an unexpected TaskCanceledException when sending request to Salesforce API even after multiple retries!,Source=Microsoft.Connectors.Salesforce,''Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib

phemanth 10,250 Reputation points Microsoft Vendor

2024-09-18T10:27:46.1266667+00:00
@LY

Thanks for posting your question in the Microsoft Q&A forum.

The error message indicates that the copy activity is failing with a TaskCanceledException when trying to copy larger datasets (around 50GB) from Salesforce using the new linked service connector. This suggests that the issue is related to the timeout or throttling of the Salesforce API.

Increase Timeout Values: Adjust the timeout settings in your Azure Data Factory copy activity to allow more time for the data transfer. This can help prevent the task from being canceled due to timeouts.

Check Salesforce Bulk API Limits: Ensure that you are not exceeding the Salesforce Bulk API limits. Salesforce Bulk API 2.0 has specific limits on the number of batches you can submit per 24-hour period. If you exceed these limits, you may encounter failures.

Optimize Data Transfer: Consider breaking down the large dataset into smaller chunks and transferring them sequentially. This can help manage the load and reduce the risk of timeouts.

Increase Node Concurrency: If you are using a self-hosted integration runtime (IR), increase the node concurrency settings to allow more parallel data transfers.

Monitor API Usage: Keep an eye on your Salesforce API usage and ensure that you have sufficient API calls available for the data transfer process.

Retry Logic: Implement retry logic in your pipeline to handle transient errors and retries automatically.

refer :https://video2.skills-academy.com/en-us/answers/questions/1669230/salesforce-v2-link-service-doenst-work-error-messa?comment=question#newest-question-comment https://video2.skills-academy.com/en-us/azure/data-factory/connector-salesforce?tabs=data-factory

Let me know if you need further help or have any other questions!
LY 0 Reputation points

2024-09-18T21:14:37.6933333+00:00

Thanks for the reply. I am not trying to copy the data into Salesforce instead i am trying to pull the data from Salesforce into a blob storage. I have checked the timeout settings for my adf pipeline and i can see it is mentioned as below. Also, the exception says, it has retried multiple times. I would like to check on Bulk API limits to retrieve the data from salesforce when using ADF. Is there any documentation on how to implement bulk api in ADF for salesforce data retrieval?
phemanth 10,250 Reputation points Microsoft Vendor

2024-09-19T11:32:05.5066667+00:00

@LY

Thanks for clarifying! Since you’re pulling data from Salesforce into Blob Storage, it’s important to understand the Bulk API limits and how to implement it in Azure Data Factory (ADF).

Salesforce Bulk API Limits

Batch Limits: You can submit up to 15,000 batches per rolling 24-hour period. Record Limits: Each batch can contain up to 10,000 records, and you can upload up to 150 million records per 24-hour period. Data Size: A bulk query can retrieve up to 15 GB of data, divided into 15 files of 1 GB each.

Implementing Bulk API in ADF

To use the Bulk API for data retrieval in ADF, follow these steps:

Configure the Linked Service:

Ensure that the Salesforce linked service in ADF is configured to use Bulk API 2.0. You can set this in the linked service properties by specifying the apiVersion property.

Set Up the Copy Activity:

In your ADF pipeline, configure the copy activity to use the Salesforce linked service. Ensure that the source settings are optimized for bulk data retrieval.

Adjust Timeout and Retry Settings:

Since your timeout is already set to 7 days, ensure that your retry settings are configured to handle transient errors effectively.

Monitor and Optimize:

Keep an eye on your API usage and batch limits. You can monitor this through Salesforce’s API usage dashboard.

For detailed guidance, you can refer to the official documentation on using the Salesforce connector in ADF.

Share via

Failing to copy large dataset from Salesforce using new salesforce V2 connector in Azure data factory

Your answer