Data Factory and bad performances with REST API GET requests (SFCC API)

Question

HI

I'm facing very long response time to acquire Salesforce commerce cloud data (via REST API)

The principle is to get a list of orders (activity lookup) and iterate on the ids by invoking an APi URL (for each activity)
And in the end to make each time a CSV file (activity copy)

We will say that there are on average 1000 orders
Response time problem appears after a while

Is there a bottleneck on the SFCC server ?
By cons I can not do anything as an adjustment with them
Many articles talk about caching API requests received
https://video2.skills-academy.com/en-us/answers/questions/34966/data-factory-foreach-performance-questions-with-re.html

I wonder if I should not put a wait activity every 20 requests for example?
there could be an sceario that the the calls in the remote servers are getting queued , but at this time its just an assumption
Have you ever faced this problem , please ?

Are well and thank you in advance

Christopher

-------------------------------------------------------------------------------------------------------------------------------------------

"inputs": [
{
"referenceName": "HTTP_JSON",
"type": "DatasetReference",
"parameters": {
"URLDynamique": "@markus.bohland@hotmail.de ('/s/XXXX/dw/shop/v19_8/orders/',pipeline().parameters.Ordernumber) "
}
}
],
"outputs": [
{
"referenceName": "BLOB_FILE_CSV",
"type": "DatasetReference",
"parameters": {
"filename": "Orderstaging.csv"
}

Answer

Hello ,
Thanks for the question and also for using this forum.

As I understand the main ask here is if the issue is on the ADF side or on the SFCC side . I think in the refereneced thread the issue ultimately was on the API side . I think I can suggest you call the API from powershell locally and check the performance . I am pating a dummy code , which should help you with some updates .

$uri = "https://URL"

AllTheOrders=Get-ChildItem -Path X:\Test

foreach ($id in $AllTheOrders) {
    $startTime = Get-Date
    Write-Host "The script was started $startTime"
    $conversation  = Invoke-WebRequest -Method GET -Uri "$uri/someAPI/$id" | ConvertFrom-Json
    $EndTime = Get-Date
    Write-Host "The script was started $EndTime"

}

This way you can capture the performance locally and compare that with the ADF one . Please do let me know how it goes . I also think that if you are adding a wait activity , then you should check the sequential option and not the parallel one .

Thanks& stay safe
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

Share via

Data Factory and bad performances with REST API GET requests (SFCC API)

1 answer

Your answer