IIS Health Checking can Prevent the Creation of a W3WP.exe Userdump
Problem
Sometimes when we're trying create a usermode dump of a w3wp.exe process something sabotages the attempt. The dmp file isn't successfully created and the dump-triggering tool balks.
When using Taskmanager to try to create the dump, the error is:
Dump Process
The operation could not be completed.
Only part of a ReadProcessMemory or WriteProcessMemory request was completed.
OK
When using DebugDiag 2.0 to try to create the dump, the error is:
Debug Diagnostic Tool
Could not create dump file *.dmp for process id *.
GetLastError returns 0x8007012B.
This scenario is most common when the w3wp.exe is large (in terms of virtual bytes) and the web server may be a bit taxed on either CPU or memory. The problem may not happen when trying to dump out small w3wp.exe processes.
Root Cause
When you're making a memory dump of a usermode process, everything in the process gets frozen in time while the dmp file is written out. While everything (all the threads) is frozen inside the w3wp.exe, IIS health checking is "pinging" the Application Pool to look for problems like unresponsiveness. If the w3wp.exe seems unresponsive, the Windows Process Application Service (WAS) will recycle that Application Pool.
The sign of this happening is in the System Event Log. Look for WAS as the source of the event and 5010 as the event ID. Event ID 5010 will say something like, "A process serving application pool . . . failed to respond to a ping. The process id was . . ."
Workaround
By default, WAS will ping the appPool every 30 seconds and gives it 90 seconds to respond. If the worker process can't respond, WAS will recycle the Application Pool. The unresponsive w3wp.exe disappears and a fresh one takes its place.
There are a few options to consider.
- You can try dumping the w3wp.exe sooner while its smaller. If you're troubleshooting a memory leak however, you may want to dump the process while its big.
- You can set Ping to disabled for the AppPool in the IIS manager. Wait for a server maintenance window, open the INETMGR, drill down to the Application Pools, find and select the AppPool that you're trying to get a dump of, select ADVANCED SETTINGS, scroll down to PING ENABLED, and change from TRUE to FALSE.
- Or change the Ping Maximum Response time to number higher than 90 and/or the Ping Period to some number higher than 30.
Comments
- Anonymous
April 24, 2014
Awesome! Thank you.