Possible Handle Leak on Azure App Service App

Krishnadas K P_FT 5

We have an ASP.NET 4.8 application hosted in Azure App Service. At times we see a spike in latency in usually a single instance which also shows spikes in CPU time, TCP connections, Thread Count and Handle count. The average thread count and and average handle count spikes from about 100 to 450 and 4000 to 8000+ respectively. We did an investigation by analysing memory dumps through windbg and could see the number of Event and Thread handles were unusually high at this time. The thread and handle count does not come down for quite a few time, but the latency is only felt right at the beginning of this spike. So we believe the CPU time and latency spikes are due to the cost of thread and handle spawning.

How can we further debug the potential handle or thread leak issue ? I am not able to find out what requests are causing the spikes in thread and handle from windbg. I read some docs but still not getting clarity on what is thread and event handles.

ajkuma 24,151 Reputation points Microsoft Employee

2023-01-25T12:08:26.43+00:00
@Krishnadas K P_FT , Apologies for the delayed response. Thank you for sharing detailed information and your observations.

I understand you have analyzed memory dumps and from the issue description you’re running multiple instances. If you haven’t done these already, you may try these steps to further narrow down the issue.

Were there any changes prior to this issue? Or has it always been the issue since WebApp deployment?

For app code standpoint: Try to avoid synchronous calling when developing .NET Core applications. Synchronous calling blocks the next execution until the current execution is completed. While fetching data from an API or performing operations like I/O operations or independent calling, execute the call in an asynchronous manner.

Avoid using Task.Waitand Task.Result, and try to use Await.

Identify how much memory is available on the instance.
Review “Metrics per instance (App Service Plan)” blade.

If you have a small instance, Scale-up and check if it makes any difference.

For additionally analysis, the dump file can obtained from Azure Portal, if the issue is currently occurring:
Azure Portal > Diagnose and Solve Problems > Diagnostic Tools > Collect Memory Dump.

If the issue happens sporadically, you may create an Auto Heal rule based on memory limit.
Azure Portal> Diagnose and Solve Problems > Diagnostic Tools > Auto-Heal

To tackle the issue automatically, you consider to set up healthy check for the App service, which will help us to remove the unhealthy instance from the plan automatically - App Service - Health Check

Kindly let us know how it goes, we would need additional info about your WebApp to gather much deeper insight of issue.
Ryan Hill 26,866 Reputation points Microsoft Employee

2023-01-28T01:32:42.3433333+00:00

@Krishnadas K P_FT please let us know if the response down below was helpful.
Marc O'Morain 0 Reputation points

2023-03-21T16:28:04.31+00:00

@Krishnadas K P_FT did you find the cause of this? I'm investigating a very similar issue myself.

1 answer

Ryan Hill 26,866 Reputation points Microsoft Employee

2023-01-26T05:06:04.6366667+00:00

@Krishnadas K P_FT I'm assuming you're already familiar with the DaaS options in the Diagnose and solve problems blade. In addition to what was mentioned above, I would suggest collecting a .NET Profiler trace.

If needed, you can extend the profiling timeout, which defaults to 60 seconds, by setting APPSETTING_IIS_PROFILING_TIMEOUT_IN_SECONDS application setting. The .diagsession file the profiler returns you can actually examine inside Perfview. I would also advise reviewing this app service blog post for more information regarding profiling your ASP.NET application.
Please sign in to rate this answer.
Krishnadas K P_FT 5 Reputation points

2023-02-03T07:04:55.1766667+00:00

Thanks for the reply, Ryan. We have already taken memory dump during the incident and analyzed it. That's how we noticed the thread and handle spike in the app. Do we get any deeper insight from the profiler trace ?

Ryan Hill 26,866 Reputation points Microsoft Employee

2023-02-11T16:36:27.8366667+00:00

It should. The dump shows the spike, but the profile should at least direct you into where the increased handles started to appear. If the issue is fairly repeatable, the profiler should be helpful in determining what's happening. If it isn't, then let me know. We'll need to work more closely with you.

Krishnadas K P_FT 5 Reputation points

2023-03-13T05:15:35.2066667+00:00

Hi Ryan,

Apologies for the delay in responding. Thankfully we are having a period of stability without this issue, but unfortunately that means I do not have an occasion to take the profile trace. I will try to reproduce the issue again elsewhere and get back to you after analyzing the trace. Thanks for the help

Banken, Joe 0 Reputation points

2023-06-05T15:17:55.63+00:00

@Krishnadas K P_FT Did you ever figure it out? I am running into the same issue.

Krishnadas K P_FT 5 Reputation points

2023-06-06T05:57:50.6533333+00:00

@Banken, Joe Nope. We had this issue re-occur once recently. All same behavior. But no luck in identifying what's going wrong. We had this more frequently while running P1V3 size but has come very rare when running P2V3
Sign in to comment

Share via

Possible Handle Leak on Azure App Service App

1 answer