Troubleshooting CreateWindowEx failures

  • [アーティクル]

Several years back, my team was handling multiple issues where Windows Forms application would throw an exception while trying to create a new form, control, etc. While the description string that Windows Forms included in the exception ("Error creating window handle.") indicated that a call to CreateWindowEx failed, the exception did not provide any information on why the call failed. At the time, I wrote a small document that described the steps for debugging native and .NET applications to determine why a given CreateWindowEx call fails. One of my colleagues suggested creating a blog post with the information from that document, so here we go.

 

The CreateWindowEx function

We should first discuss how the CreateWindowEx function works before diving into how to troubleshoot CreateWindowEx failures.Windows applications, including .NET applications that use Windows Forms or Windows Presentation Foundation, create their windows by calling the CreateWindowExA or CreateWindowExW functions. Both functions internally call a common USER32 function that will perform some parameter validation, such as the window styles and handles specified in the call, handle creating a MDI child window if the WS_EX_MDICHILD extended window style is specified and processes the current activation context for the calling thread. If all is well on the USER32-side of the CreateWindowEx call, it then calls into the kernel-mode (WIN32K) implementation of CreateWindowEx.

The kernel-side of the CreateWindowEx call will first check the following conditions:

  • Does the specified window class exist?
  • Can the text for the atom be retrieved, if the caller specified an atom for the class name?
  • Was a parent window specified, if creating a child window?
  • Will the new window exceed the nested window depth limit, if creating a child window or an owned window? The default limit is 50 levels deep and can be configured via the USERNestedWindowLimit registry value.
  • Does the parent/owner window belong to the same desktop as the calling thread?

Assuming that each of these checks pass, CreateWindowEx will perform the following tasks when attempting to create a new window object:

  • Determine if creating a handle for the new window object will exceed the User handle quota limit for the calling process.
  • Allocates memory for the new window object from the desktop's heap.
  • Initializes the memory for the new window object
  • Creates a handle for the new window object in the User handle table

Assuming that the window object was created successfully, CreateWindowEx will notify registered WinEvent and WH_CBT hooks that a new window object has been created, determine the location, size, z-order position, etc. of the new window, followed by calling the window's WndProc to process WM_NCCREATE and WM_CREATE window messages. CreateWindowEx then returns the handle of the new window object.

 

Causes of CreateWindowEx failures

You may have noticed some of the conditions that will cause CreateWindowEx to fail from the previous section. Let's enumerate conditions that will cause CreateWindowEx to fail:

  • The specified window class does not exist.
  • Using invalid window styles or extended window styles.
  • Using invalid User handles, such as window handles and menu handles.
  • Attempting to create a child window without specifying a parent window.
  • Attempting to create a child window or an owned window and the specified parent/owner belongs to a different desktop than the calling thread.
  • Creating a child or owned window will exceed the nested window limit.
  • Creating a new window object will exceed the handle quota for the calling process.
  • There is insufficient heap available in the desktop’s heap to allocate memory for the new window object.
  • There are no available entries in the User handle table.

Note that while a new window object is successfully created, the CreateWindowEx function may still return NULL. How can this happed?

Recall that before returning, CreateWindowEx will call the window's WndProc to process a WM_NCCREATE and a WM_CREATE window message. The WndProc can stop the window creation by returning 0 when handling the WM_NCCREATE window message or by returning -1 when handling the WM_CREATE window message. CreateWindowEx handles this situation by destroying the window object and returning NULL. Additionally, a WH_CBT hook can stop the window creation by returning 0 from its hook procedure when handling the HCBT_CREATEWND notification.

 

CreateWindowEx failures and GetLastError

The Windows SDK documentation for the CreateWindowEx function states that extended error information can be obtained by calling GetLastError when CreateWindowEx returns NULL. But how useful is the error code returned by GetLastError?

The answer is that it depends on the error condition that caused CreateWindowEx to return NULL.

For example, GetLastError will return ERROR_CANNOT_FIND_WND_CLASS if the window class does not exist. On the other hand, GetLastError may return NOERROR or some error code that seems unrelated to calling CreateWindowEx if the window's WndProc stopped the window creation when handling the WM_NCCREATE or WM_CREATE messages.

 

Troubleshooting CreateWindowEx failures

Although GetLastError does not always return a valid error code on a CreateWindowEx failure, you should always check the error code as it may help diagnose the source of the failure. Current versions of the .NET Framework included the value returned by GetLastError when throwing a Win32Exception. Earlier versions of the .NET Framework did not include the error code when throwing a Win32Exception.

Hitting the User handle quota limit for the process should be pretty simple to identify. You can view the User object handle count for a process in Task Manager (taskmgr.exe) or by calling the GetGuiResources function. If the handle count is approaching the User handle quota limit, then Task Manager and GetGuiResources is likely reporting a value near the limit. Additionally, GetLastError should return ERROR_NO_MORE_USER_HANDLES. The default limit is 10,000 and can be configured via the USERProcessHandleQuota registry value.

Running out of desktop heap is a little more difficult to identify since there is no mechanism that ships with Windows to measure available desktop heap for a given desktop. However, GetLastError should return ERROR_NOT_ENOUGH_MEMORY. You can find more information on desktop heap at blogs.msdn.com/b/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx.

Then there are the scenarios where CreateWindowEx fails and GetLastError returns NOERROR or what appears to be an invalid error code. Generally, in this scenario, the failure is caused by the window's WndProc stopping the window creation. There are some user-mode troubleshooting options that you can use before you start stepping through the disassembly of the kernel side of the CreateWindowEx function.

One approach is to set a WH_CBT window hook on the thread calling CreateWindowEx to determine if a window object has been created for a given CreateWindowEx call. Recall that WH_CBT hook procedures are called with the HCBT_CREATEWND notification after a window object has been created and before the window procedure is called to handle the WM_NCCREATE and WM_CREATE window messages.

You can then set a WH_CALLWNDPROC window hook on the thread calling CreateWindowEx or set a breakpoint on the window's WndProc (if known) to determine if the window's WndProc is called to handle the WM_NCCREATE and WM_CREATE window messages. You can also use the Spy++ tool included with Visual Studio to monitor when windows owned by a thread process WM_NCCREATE and WM_CREATE window messages.

 

Debugging CreateWindowEx with WinDBG

I tend to use the Debugging Tools for Windows for most of my debugging work, specifically WinDBG. The debugging tasks described below are from a 32-bit Windows 7 system using the version of WinDBG included with the Windows SDK for Windows 7 and the public Microsoft symbol server.

You may encounter a scenario where you are trying to troubleshoot a CreateWindowEx failure where you do not know the address of the window's WndProc, such as a third-party control. How do you set a breakpoint on the WndProc?

Recall that most of the work for creating a window is done in kernel mode, which includes calling the window's WndProc to process WM_NCCREATE and WM_CREATE window messages. Since the kernel side of CreateWindowEx tries to call a user mode function, Windows needs to transition back into user mode to call the WndProc, which is done via the DispatchClientMessage function.

You can use a breakpoint on CreateWindowEx that enables a breakpoint on DispatchClientMessage and another breakpoint when CreateWindowEx returns that disables the breakpoint on DispatchClientMessage. This should allow you to determine if the window procedure of the window is being called.

Here are breakpoints that I used when debugging an instance of Notepad:

0:000> bl
 0 e 75934ec3 0001 (0001) 0:**** USER32!DispatchClientMessage ".echo ***** DispatchClientMessage *****;dd @esp+4 L4;g"
 1 e 7592ec7c 0001 (0001) 0:**** USER32!CreateWindowExW ".echo ***** CreateWindowExW called *****;be 0;g"
 2 e 7592ecb0 0001 (0001) 0:**** USER32!CreateWindowExW+0x34 ".echo ***** CreateWindowExW returned *****;bd 0;r @eax;g"

Breakpoint 0 is initially disabled. It will be enabled when CreateWindowEx is called and disabled when CreateWindowEx returns. When hit, the breakpoint will dump the parameters passed to the function and then resume execution. You could also set up the breakpoint to halt in the debugger where you could then step into the window procedure. Note in the debug output below that the last parameter to DispatchClientMessage is the address of the window procedure.

Breakpoint 1 displays a message that CreateWindowEx is being called, tries to display the class name passed to CreateWindowEx (which will fail if an atom is passed) and enables the breakpoint on DispatchClientMessage.

Breakpoint 2 displays a message that CreateWindowEx is returning, displays the value of the handle that it is returning and disables breakpoint 0.

Below is the debugger ouput from the first CreateWindowEx call in the process:

***** CreateWindowExW called *****
***** DispatchClientMessage *****
001bf504 00b9fc18 00000081 00000000 001bf56c
***** DispatchClientMessage *****
001bf548 00b9fc18 00000083 00000000 001bf594
***** DispatchClientMessage *****
001bf4d4 00b9fc18 00000001 00000000 001bf53c
***** DispatchClientMessage *****
001bf554 00b9fc18 00000005 00000000 00000000
***** DispatchClientMessage *****
001bf554 00b9fc18 00000003 00000000 00000000
***** CreateWindowExW returned *****
eax=000602fe

Note the window messages that are being received by the window. WM_NCCREATE is 0x81 and WM_CREATE is 0x1. The other messages are also a part of normal window creation.

In the example above, you might have noticed that I skipped how I found the address of the return from CreateWindowEx in order to set breakpoint 2. The method that I used was to examine the disassembly of the CreateWindowExW function and looked for the return instruction, highlighted below.

0:000> u user32!CreateWindowExW L13
USER32!CreateWindowExW:
7592ec7c 8bff mov edi,edi
7592ec7e 55 push ebp
7592ec7f 8bec mov ebp,esp
7592ec81 6800000040 push 40000000h
7592ec86 ff7534 push dword ptr [ebp+34h]
7592ec89 ff7530 push dword ptr [ebp+30h]
7592ec8c ff752c push dword ptr [ebp+2Ch]
7592ec8f ff7528 push dword ptr [ebp+28h]
7592ec92 ff7524 push dword ptr [ebp+24h]
7592ec95 ff7520 push dword ptr [ebp+20h]
7592ec98 ff751c push dword ptr [ebp+1Ch]
7592ec9b ff7518 push dword ptr [ebp+18h]
7592ec9e ff7514 push dword ptr [ebp+14h]
7592eca1 ff7510 push dword ptr [ebp+10h]
7592eca4 ff750c push dword ptr [ebp+0Ch]
7592eca7 ff7508 push dword ptr [ebp+8]
7592ecaa e8edfeffff call USER32!_CreateWindowEx (7592eb9c)
7592ecaf 5d pop ebp
7592ecb0 c23000 ret 30h

I then use the following syntax to set the breakpoint:

bp2 7592ecb0 ".echo ***** CreateWindowExW returned *****;bd 0;r @eax;g"

Alternately, you can set a conditional breakpoint when CreateWindowEx returns and use the !gle command to dump the value that will be returned by GetLastError if CreateWindowEx fails, where the EAX register is zero. In the scenario below, I created a simple Windows Forms application that repeatedly creates child windows until the User handle process quota is reached and CreateWindowEx fails.

I use the following command to set a conditional breakpoint on the return from CreateWindowExW that breaks in the debugger when the function fails:

bp 7592ecb0 "j (@eax==0) '.echo ***** CreateWindowExW failed. *****';'g'"

Once the debugger breaks on a CreateWindowEx failure, you can obtain the last error value with the !gle command:

0:000> !gle
LastErrorValue: (Win32) 0x486 (1158) - The current process has used all of its system allowance of handles for Window Manager objects.

You can also execute other debugger commands, including .NET debugger extension commands such as !clrstack:

0:000> .loadby sos clr
0:000> !clrstack
OS Thread Id: 0xea4 (0)
Child SP IP Call Site
001ae9c4 7592ecb0 [InlinedCallFrame: 001ae9c4]
001ae9a8 5e6c50ef DomainBoundILStubClass.IL_STUB_PInvoke(Int32, System.String, System.String, Int32, Int32, Int32, Int32, Int32, System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, System.Object)
001ae9c4 5e6a978c [InlinedCallFrame: 001ae9c4] System.Windows.Forms.UnsafeNativeMethods.IntCreateWindowEx(Int32, System.String, System.String, Int32, Int32, Int32, Int32, Int32, System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, System.Object)
001aea88 5e6a978c System.Windows.Forms.UnsafeNativeMethods.CreateWindowEx(Int32, System.String, System.String, Int32, Int32, Int32, Int32, Int32, System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, System.Object)
001aead0 5e6a8e9f System.Windows.Forms.NativeWindow.CreateHandle(System.Windows.Forms.CreateParams)
001aeb54 5e6a8b82 System.Windows.Forms.Control.CreateHandle()
001aebac 5e6a8921 System.Windows.Forms.Control.CreateControl(Boolean)
001aebe4 5e6a8818 System.Windows.Forms.Control.CreateControl()
001aebfc 5e6b1c31 System.Windows.Forms.Control+ControlCollection.Add(System.Windows.Forms.Control)
001aec38 5e6b1aab System.Windows.Forms.Form+ControlCollection.Add(System.Windows.Forms.Control)
001aec4c 5ef8f027 System.Windows.Forms.Control.set_ParentInternal(System.Windows.Forms.Control)
001aec58 5ebfd59a System.Windows.Forms.Control.set_Parent(System.Windows.Forms.Control)
001aec60 002903c3 WindowsFormsApplication1.Form1.button1_Click(System.Object, System.EventArgs)*** WARNING: Unable to verify checksum for WindowsFormsApplication1.exe
 [D:\Cases\WindowsFormsApplication1\WindowsFormsApplication1\Form1.cs @ 24]
001aec80 5e654507 System.Windows.Forms.Control.OnClick(System.EventArgs)
001aec94 5e656ca2 System.Windows.Forms.Button.OnClick(System.EventArgs)
001aecac 5ec3a480 System.Windows.Forms.Button.OnMouseUp(System.Windows.Forms.MouseEventArgs)
001aecc8 5ec03dd1 System.Windows.Forms.Control.WmMouseUp(System.Windows.Forms.Message ByRef, System.Windows.Forms.MouseButtons, Int32)
001aed5c 5efa6a2f System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
001aed60 5efae391 [InlinedCallFrame: 001aed60]
001aedb4 5efae391 System.Windows.Forms.ButtonBase.WndProc(System.Windows.Forms.Message ByRef)
001aedf8 5e6c19f8 System.Windows.Forms.Button.WndProc(System.Windows.Forms.Message ByRef)
001aee04 5e6aa393 System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
001aee0c 5e6aa311 System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
001aee20 5e6aa256 System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
001aefc4 00840ab8 [InlinedCallFrame: 001aefc4]
001aefc0 5e6c6c1c DomainBoundILStubClass.IL_STUB_PInvoke(MSG ByRef)
001aefc4 5e6ba99f [InlinedCallFrame: 001aefc4] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
001af008 5e6ba99f System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
001af00c 5e6ba5cc [InlinedCallFrame: 001af00c]
001af0a4 5e6ba5cc System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
001af0fc 5e6ba421 System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
001af12c 5e642815 System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
001af140 002900ae WindowsFormsApplication1.Program.Main() [D:\Cases\WindowsFormsApplication1\WindowsFormsApplication1\Program.cs @ 18]
001af370 710d21bb [GCFrame: 001af370] 

Conclusion

Hopefully you will be able to apply some of the techniques described in this post if you find yourself in the situation where you are trying to troubleshoot CreateWindowEx failures in managed and native applications.

Comments

  • Anonymous
    February 07, 2013
    At my work we have the situation that the CreateWindowEx function hangs, it does not returns a handle, it does not return an error. Just waiting. Only after logging of the pc (in that process) you can see the applications pops up. Do you have an idea what the can be the cause of that?