How to configure IAG to use AES 256 encryption

By default client browsers (or at least, any reasonably up to date client browser) will connect to IAG using 128 Bit encryption. This can be seen by right clicking in the browser pane and choosing ‘Properties’ after you have accessed your IAG portal. For example:

Internet Explorer 7 on Windows XP:

Internet Explorer 8 on Windows Vista:

To view the encryption in use on Firefox the process is similar. Right click in the browser pane and choose ‘View Page Info’, then click the ‘Security’ padlock icon. Here is a screenshot of Firefox 3.0 on Windows XP:

128 bit encryption is pretty strong and is adequate for most use. But what if you have a requirement for stronger encryption? Can IAG support 256 bit encryption? This blog entry would be rather pointless if the answer was ‘No’ so clearly it can. Let’s see how we can configure this – it’s pretty easy.

Step 1: Add support for AES 128 and AES 256 to the OS.

When Windows Server 2003 was released it supported up to 128 bit symmetric encryption algorithms for SSL. Whilst Windows Server 2003 supported AES It did not have any support for the AES cipher suites in schannel.dll. An update has subsequently been released that adds AES128 and AES 256 cipher suite support into schannel.dll for Windows Server 2003. So our first step is to install this update on the IAG server. The update can be downloaded from here.

Install it on IAG and restart your IAG server for the new cipher suites to become available for use.

If you test from your browser clients you will see some differences now:

• Windows XP with IE7 still uses 128bit RC4
• Windows Vista with IE8 now uses 128 bit AES. As does Firefox 3.

So we’re getting closer, but still no 256 bit encryption. It appears that even though AES 256 is an option, it is not offered as the preferred cipher, presumably for performance reasons (256 bit encryption will take a bit more effort to encrypt/decrypt than 128 bit encryption).

Step 2: Forcing AES 256

On the IAG server open regedit.exe and browse down to:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel\Ciphers

Here you will find details of the all ciphers available to Schannel (which is responsible for SSL/TLS)

To stop Schannel from offering AES 128 open the ‘AES 128/128’ key and create a REG_DWORD value ’Enabled’ with the hexadecimal value ‘0’. Then open the ‘AES 256/256’ key and create a REG_DWORD value ’Enabled’ with the hexadecimal value ‘C0’. This value enables AES 256 for TLS1 (the AES algorithms will not work with SSL3).

If you want to block everything apart from AES 256 then set ‘Enabled= 0’ for each of the other ciphers. You can achieve the same thing by opening the IAG configuration UI, then from the toolbar choosing ‘Admin/SSL Configuration’ and unchecking all the Symmetric Ciphers. Activate the IAG to update the registry. Whichever way you use, a reboot is required to make the new settings take effect.

Restart the IAG again and you are done

Now let’s try again with our browsers

Windows XP with IE7 still uses 128bit RC4 (or will fail to connect if you disabled everything except AES 256)

Windows Vista with IE8 and Windows XP with FF3 both now use AES 256

So what Browser/OS combinations can you use to get AES 256 support?

• IE7 upwards on Windows Vista (or Windows 7).
• Firefox on XP or Vista (I’ve not tested on Windows 7)

I’ve not tested Chrome, Safari, Opera or other browsers (or other non-Windows operating systems) so support may vary. But once you have configured IAG to offer AES256 it should be straightforward to test for any other browsers that you need to support.

Just remember that switching to 256 bit AES encryption is likely to take more processing power to handle than 128 bit RC4 encryption, (though it may actually be a better choice than 168bit 3DES) and being an SSL VPN, obviously IAG uses SSL extensively. Before choosing to use 256 bit encryption remember that you are gaining security at the expense of performance. Make sure you have the balance right for your organization and test to make sure performance and throughput are acceptable.

Author

Phil Bevan

Support Engineer – IAG Team

Microsoft - UK

Tech Reviewer

Noam Ben-Yochanan

Senior Program Manager – IAG Product Team

Comments

• Anonymous
  October 13, 2009
  Do any of the other encryption controls in the OS effect IAG? For example the local security policy:  secpol.msc "System cryptography: Use FIPS compliant algtorithms for encryptio, hashing, and signing." That control is supposed to restrict the system to using the FIPS 140 crypto module.  Schannel is supposed to use this restriction.  However if you set that policy without this hotfix then you will still get RC4 which is not FIPS in any keylength. Installing this patch makes IAG negotiate AES 128 which is FIPS compliant even if the "use FIPS" local policy is not enabled. What I really want to know is if installing this patch results in IAG using the FIPS certified crypto service provider?