A Slippery System.IO.FileLoadException

After installing the framework 3.5 SP1 (kb 951847) (which also updates framework 2.0) an asp.net web app occasionally began throwing the following exception, as reported by their custom error page:

An error occurred attempting to process your request. Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401) MyCustomDll.MyCustomDllat MyCustomDll.MyCustomDll.

This should be a System.IO.FileLoadException, but there never seemed to be any record of the System.IO.FileLoadException.  It didn't even show up in the application event log as expected:

Event Type:     Warning
Event Source:  ASP.NET 2.0.50727.0
Event Category:          (3)
Event ID:          1309
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Process name: w3wp.exe
Exception type: Exception
Exception message: An error occurred attempting to process your request.
Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)
 
Request information:
    Request URL: <https://www.mysite.com:443/Application/Jobs/CustomStuff.aspx?
    Request path: /APPLICATION/Jobs/CustomStuff.aspx
Thread information:
   Stack trace:    at MyCustomDll.MyCustomDll.CustomStuff.updateButton_Click(Object sender, EventArgs e)
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
 

Getting the dump of the w3wp.exe at the right time was impossible at first. We couldn't dump on CLR exception with a DebugDiag 1.1 crash rule because instead of the expected System.IO.FileLoadException was showing up as a ',' rather than a 'System.IO.FileLoadExeception' according to the debugdiag crash rule log file.  It continued to show up as a “blank exception” in the debugdiag log.txt files (made by a crash rule) until the customer installed the updates-to-3.5-sp1 (KB959209 = KB958481/KB958483/KB958484).  After those updates Debugdiag was able to trip a dump when the System.IO.FileLoadException was thrown:

Exception type: System.IO.FileLoadException
Message: Loading this assembly would produce a different grant set from other instances. (Exception from HRESULT: 0x80131401)
InnerException: <none>

The dump showed that the web application was initializing a class in a GAC'd assembly ("C:\WINDOWS\assembly\GAC\MyCustomDLL\1.0.530.2__26fddaf73a962ee0\MyCustomDLL.Integration.Export.dll") and also showed that somehow this assembly has a different "SecurityDescriptor" in the "Shared Domain."  The calling assembly MyCustomDLL.Base.DLL is not present in GAC and so it gets loaded in LoadFrom loader context. Now when calling assembly tries to load the assembly MyCustomDLL.Integration.Export.dll, fusion sees that it is GACed and  already loaded in Load Context as a domain neutral assembly.

So far I know of two workarounds to this issue:

1)   Remove the called assembly MyCustomDLL.Integration.Export.dll and its dependency MyCustomDll.Data.dll from GAC so that it does not get loaded as domain-neutral in load context
 

or

2)   Move the calling assembly to GAC so that it also gets loaded in Load Context.