SCOM 2012 WebConsole and FIPS Compatibility

Disclaimer: Due to changes in the MSFT corporate blogging policy, I’m moving all of my content to the following location. Please reference all future content from that location. Thanks.

Updated article on this subject can be found here. It has SCOM 2016 info, as well as a link to the a download for the files you need.

As a quick overview, FIPS stands for Federal Information Processing Standards and it’s a set of algorithms that has to do with encryption of data at rest and data in transit.  FIPS is not turned on by default as many applications are not FIPS compliant out of the box.  You see this in use more in high security environments.  Like many applications, SCOM is not FIPS compliant by default.  Changes need to be made in order for it to be compliant.  There’s a KB article on this subject published here.  This article changed at some point and it’s currently inaccurate in a few places and somewhat misleading.  I’ve placed a request to get it updated, but to date it has not been updated.  Updating for FIPS is a somewhat cumbersome process, so I’ll walk through the step by step in this blog.

First off, let’s determine if FIPS is enabled.  That’s easy enough to do.  Open the registry and navigate to the following key:

HKLM\System\CurrentControlSet\Control\Lsa\FIPSAlgorithmPolicy

The “Enabled” DWORD value will be set to 1. This is often set via GPO in secure environments, so simply changing this will be reverted. 

Turn on FIPS and reboot your system, and your webconsole will look something like this:

I would note that on some internal cases, there can be an internal server error listed here instead.

Now to the fix:  WARNING: Backup all files you touch in this section before proceeding.

Step 1:   The first thing you need to do is to get a DLL file off of the SCOM CD. IMPORTANT NOTE: This file is found only on the SCOM 2012 RTM or SCOM 2012 with SP1 CD.   It is not on the SCOM 2012 R2 CD.  They are located in the \SupportTools\AMD64 folder of the CD.  Copy the Microsoft.EnterpriseManagement.Cryptography.dll file to C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server.  (note, this is your SCOM installation directory.  This is the default location, but that can vary based on your setup).

Also download a copy of gacutil.exe.  This isn’t easily found.  It’s a part of the SDK kit for Windows as well as some of the .NET development tools.  It is not in the SCOM CD.  I’d make it available for download, but that doesn’t seem to be an option for this source.   The SDK kit is free, and can be found here.  It’s a rather large download, so be patient.  For the purposes of this step, place gacutil.exe on your desktop.

Step 2:   Next, we will open up a command prompt in administrative mode and then type the following:

cd desktop and press ENTER

gacutil.exe /i “C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Microsoft.EnterpriseManagement.Cryptography.dll” and press ENTER.

Tope notepad and press ENTER.

What you’ve done to install the DLL file and then open up notepad in admin mode.  You want this because the files you need to edit are in protected locations on the OS, and as such, it will not let you save over them.  Opening notepad as an administrator gets you around that problem.

Step 3:   From notepad, open the following file:

C:\Windows\Microsoft.NET\Framework\v2.0.50727\CONFIG\machine.config

NOTE:  You’ll need to switch from viewing only text files to all files to see this in notepad.  As well, if your windows directory is not the standard directory, you'll need to adjust the path.  The location of where this addition should be is not specified in the article… BUT, location DOES matter. Scroll down to the very end.  Paste this information right before the </configuration> line.

Note, you may want to copy this off of the web link as my formatting isn’t quite the same.  It should look something like this:

Save it.  Do not close Notepad.

Step 4: Repeat step 3 for the following file locations:

• %WinDir%\Microsoft.NET\Framework\v4.0.30319\Config\machine.config
• %WinDir%\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config

Step 5: From notepad, open the following file:

C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\WebConsole\WebHost\web.config

This file is a bit trickier to edit due to the fact that the tags you are looking for show up twice.  In every case, the tag shows up first in a commented out section.  If you make the edits there, it won’t work.  You have to find the second tags (see screenshot below).  First locate the <encryption> tag and add the following line underneath it:

<symmetricAlgorithm iv="SHA256"/>

Second, locate the <Connection element.  Replace the <Connection… and the <Session lines with the following lines:

<connection autoSignIn="True" autoSignOutInterval="30">

<session encryptionKey="SessionEncryptionKey" tokenAlgorithm="SHA256">

Save this.  It will look something like this (note I circled the comment lines so you can see them as well).

Step 6: From notepad, open the following file:

C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\WebConsole\MonitoringView\web.config

Note this is also where the KB article goes wrong, pointing you to some .NET locations.  Those tags do not exist in those locations. Once this open, you can repeat step 5 covering the <encryption> element and <connection elements.  Those steps do not change.  There is one caveat here though.  The <session> tag is closed differently between this file and the one in step 5.  If you replace these directly, you’ll remove the ‘/’ tag at then end which will cause a mal-formed URL or something like that.  You can simply add a line and add a </session> immediately following the <session encryptionKey="SessionEncryptionKey" tokenAlgorithm="SHA256"> line.   The issues with the comment marks are present here as well.  Make sure you update the correct elements.

After that, you need to locate the <system.web> element and paste the following lines beneath it:

<machineKey validationKey="AutoGenerate,IsolateApps" decryptionKey="AutoGenerate,IsolateApps" validation="3DES" decryption="3DES"/>

Save this and close notepad.

It wouldn’t hurt to reset IIS after this point or just reboot the system.

Relaunch the web console.  It should look something like this:

If you made a mistake in the above steps, the error message should tell you where the mistake is, even so far as going to the exact line/character of the file.  You may need to turn on debugging to get that information.  If so, the error message will give you instructions on how to do that.  Compare that closely to the information in the link to see what went wrong.  Any typo can kill this.  It’s best to copy and paste where possible.

Technorati Tags: FIPS,SCOM 2012,WebConsole,Internal Error,Web Console