Getting lots of Task Canceled Exception when using Azure Search .Net SDK to merge or upload documents asynchronously

SONIA 1 Reputation point
2020-07-15T05:28:18.297+00:00

We are using Azure Search .Net SDK(v3.0.4) to merge or upload documents to one of the azure search index.

We are using the below code for the same:

var batch = IndexBatch.MergeOrUpload(documentList);
await searchIndexClient.Documents.IndexAsync(batch);

We are indexing 2 types of subindexes using the same search client. When there is a high load, we are facing lots of task cancelled exceptions and the operation is failing without any hint to inner exceptions. Can someone help me here why this issue might be there?

Stacktrace for reference: System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Rest.RetryDelegatingHandler.<>c__DisplayClass11_0.<b__1>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Rest.RetryDelegatingHandler.d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Search.DocumentsOperations.d__23.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.Azure.Search.DocumentsOperationsExtensions.d__13`1.MoveNext() --- End of stack trace from previous location where exception was thrown ---

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
831 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. ajkuma 24,396 Reputation points Microsoft Employee
    2020-07-16T11:02:56.147+00:00

    @Sonia-2755, Welcome to Microsoft Q&A! Thanks for posting a good question.

    As Bruce mentioned on your SO thread, the issue could be that you're pushing the service beyond its limits and it's not able to keep up with the request rate. We will follow up with you offline.

    Apologies for any inconvenience with this issue. Thanks for your co-operation!

    0 comments
  2. ajkuma 24,396 Reputation points Microsoft Employee
    2020-07-23T07:49:50.407+00:00

    Adding the solution that helped to benefit the community, from our offline discussions.

    Issue could be due to starvation of TCP connection pool and adding the connection settings showed improvements. To resolve other exceptions while calling Search service (Task cancelled exceptions, Http Socket exceptions, IndexBatch exceptions, Null reference exceptions, Service too busy exceptions etc), additional modifications/fix were made as per the requirement:

    1. Increase the service replicas.
    2. Decrease load on Search service by reducing calls.
    3. Adding delays.
    0 comments