Disable DEP on applications

  • [アーティクル]

Windows vista wont Allow ATL control on the machine. We have Data Execution Prevention to stop Executing ATL control on Vista.

Workaround

Add This Post Build Event

call "$(DevEnvDir)..\tools\vsvars32.bat"
editbin.exe /NXCOMPAT:NO $(TargetPath)

However if you Disable DEP for your Winform Application and than create Visual studio project by Adding Project output to your Setup Project the EXE inside MSI would have DEP enabled again.

Because It was observed in a Setup project with Visual Studio 2008 that the Add Project Output source path Points to c:\App\OBJ\*.exePost Build Event would update c:\app\BIN\*.exe and not the OBJ.

Manually add the build in setup and deployment Project
Create New Setup Project | Add File | select Build EXE which is under Bin Folder

Details

In the header of a PE file there is a flag called IMAGE_DLLCHARACTERISTICS_NX_COMPAT. This flag affects whether or not the OS enables DEP for a process. Setting this flag tells the OS that the image is compatible with DEP. For executable images, if this flag is set, the process is run with DEP enabled unless the machine is configured with the DEP policy set to AlwaysOff. If the image is a DLL and the flag is set, the OS skips checking the DLL against a compatibility database which results in a small performance improvement. All of this applies to x86, 32-bit processes only. On a 64-bit OS, DEP is always enabled for 64-bit processes, but 32-bit processes are configured by the PE flag and system policy as described above. So how does one control the flag in the PE header?

Since the C# compiler emits PE files which are MSIL only and therefore compatible with DEP, the output binaries from the VS 2008 and .NET 3.5 C# compilers have this flag set. Our expectation is that the vast majority of C# executables produced by these compilers will be part of a DEP-compatible application. For that reason we did not surface a compiler switch to configure the NXCOMPAT setting. Of course you can write a C# application that uses a native or mixed binary which is not compatible with DEP. Some ATL types in 7.1 and earlier used to do simple code generation into data pages which is a DEP no-no. If your application is generating IP_ON_HEAP exceptions, then you may need to clear the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this you can use EDITBIN.EXE from the VC toolset like so:

editbin.exe /NXCOMPAT:NO <your binary>

If you're using Visual Studio, you can add a post build step to your executable's project. You'll need to setup the environment so that EDITBIN's dependencies can be resolved. Since the post build steps you author in Visual Studio's properties page are written into a batch file that is launched by the build process, you can use Visual Studio's VSVARS32.BAT to establish the right environment. My post build step looks like this:

call $(DevEnvDir)..\tools\vsvars32.bat
editbin.exe /NXCOMPAT:NO $(TargetPath)

If you sign the binary in Visual Studio, flipping the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag in the post build step after the binary has been signed will result in an assembly that will fail strong name validation.

Comments

  • Anonymous
    December 18, 2010
    Hi.  When you say below, what do you exactly mean?  How do I do that?  Are you saying I need to create "another" setup project? "Manually add the build in setup and deployment Project Create New Setup Project | Add File | select Build EXE which is under Bin Folder"

  • Anonymous
    April 11, 2011
    If the setup projects works with the OBJ folder, isnt it just easier to just copy the output to the obj folder as well. This can be done directly after running the editbin.exe, like so(you may have to remove the x86, if you choose a different platform): ECHO Copying assembly from bin back to obj directory ECHO CALL COPY "$(TargetFileName)" "....objx86$(ConfigurationName)$(TargetFileName)" CALL COPY "$(TargetFileName)" "....objx86$(ConfigurationName)$(TargetFileName)"