Session Loss due to Application Domain Recycle-Advanced

In the previous blog we were discussing about the basics of app domain recycling and the steps we can take to confirm if app domain recycling is happening. We might come across a situation where from the steps explained previously we will come to know that app domain recycling is happening but we are not sure what triggered the app domain recycling. In this blogs we will discuss about the advanced debugging techniques to identify that.

Below are some of the data that we need to collect.

Steps to collect Procmon:

1. Download process monitor from https://technet.microsoft.com/en-us/library/bb896645.aspx

2. Run procmon.exe

3. Wait till the issue reoccurs and after the issue reoccurrence please stop the trace.

Steps to Create Debug Diag Rule:

1. We will use Debug Diagnostic 2.0 update 2 version to troubleshoot this issue further. Please install the Debug Diagnostic 2.0 update 2 tool fromhttps://www.microsoft.com/en-us/download/details.aspx?id=49924 (Please un-install all other versions of Debug Diagnostic tool before installing the latest version)

2. Before proceeding with the rule creation we need to download the symbols needed for the automated rule creation. Take the below steps to download the symbols for your w3wp process.

3. Create a Crash Rule in Debug Diagnostic 2.0 with the following steps:

a. Open DebugDiag (Start -> Programs -> Debug Diagnostic Tool 2.0 Collection)

b. Select "Crash" and click Next

c. Choose the option A specific IIS application pool and click Next and choose the application pool that is crashing from the list and click Next. (Note: The Application pool list may be empty if the IIS 6 Metabase Compatibility is not installed. In such a case you can always type the name of the application pool and click Next)

e. Click Next in Advanced Configuration (Optional), click on Breakpoints and then click on Add Breakpoint

f. If it is a .net 4.0/4.5 app then Add clr!AppDomainNative::Unload and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok. If it is a .net 2.0/3.5 app then Add mscorwks!AppDomain::UnloadWait and change the Action Type to Full User Dump and Action Limit to 5 and click on Ok.

g. Select Action Type as Full User Dump.

h. Click on Save and Close

i. Click Next for "Rule Name". The "User dump Location" can be changed here.

j. Select "Activate the rule now" and click Finish

3. Notice the Status is Active. The User dump Count will increase each time a dump file is created.

4. Wait for the issue to happen.

If everything goes well you will process monitor trace and the process dump at the time of the issue.

How to analyse process monitor trace?

As we had discussed earlier we know that application domain recycles whenever there is some change to either the framework directory or to your application directory. So we will be interested to look what exactly is getting changed at the time of the issue and who is the culprit doing it.

I will show you some of the shortcuts that you can use to analyse this trace.

As I said most likely we will be interested in the writes to any directory.

Step 1: Go to Tools->Count Occurrences->Select Operation and then click on Count.

Step 2: We will be interested in WriteFile as below. Click on Filter. Click on close.

Step 3: Next we are interested in changes to either the framework directory or application directory. Go to Filter->Filter.

Step 4: In the first drop down select Path and in second drop down select contains and in the 3 text box, type in your application directory. For example in my case it would be C:\inetpub\wwwroot\TestUserIdentity452. Click on Add

Step 5: You will get a list of write file attempts as below.

Now you can double click on any one of the above by matching the time stamp of the app domain recycle in the event log. Here you will see which process is trying to write the file and the account used by the process.

Repeat from step 4 and try the above steps for the path contains “Framework” and see if there are any writes happening.

How to analyse the dumps captured using debug diag?

Sometimes I have seen that just running debug diag analysis will give you information on the app domain unload exception. If not I will discuss some tricks as to how we can debug this using windbg.

WinDBG basic setup instructions:

If you don’t have windbg you can go ahead and install windbg from https://msdn.microsoft.com/en-us/windows/hardware/hh852365.aspx

Select and download Standalone debugging tools for windows specific to your operating systems.

You can setup the symbols for your debugger or windbg by opening windbg and going to File->Symbol File Path and entering the below path

Prerequisites:

Download the netext.dll from https://netext.codeplex.com/ which will be using for analysis.

Copy the appropriate version dll’s inside your debugger installation folder.

How to analyse the dumps?

Open the dump in debugger.

Load the netext.dll in the debugger by typing in the below command.

Before you can run any commands you need to run indexing for the dumps you have.

I have worked on 3 of the scenarios which might cause application domain recycles and below are the commands and the outputs you might get

Scenario 1: configuration change

Scenario 2: Bin directory change

Scenario 3: Page change

Reference: https://netext.codeplex.com/#wfrom

Hope this helps J