IIS6: "Supposed" Metabase Corruption

  • [アーティクル]

Prologue (required boredom)

I recently worked on an issue where the IIS Admin service would not start after a reboot. The customer had just finished installing a set of patches, and thought that might be related (we'll soon see that it wasn't related). The error was:

Event Type: Error
Event Source: Service Control Manager
Event Category: None
Event ID: 7024
Date: xx/xx/2005
Time: xx:xx:xx PM
User: N/A
Computer: xxxxx
Description:
The IIS Admin Service service terminated with service-specific error 2148073478
0x80090006).

After much initial troubleshooting by more than enough people, metabase XML files were backed up, renamed, re-copied, scrubbed, analyzed, scrutinized, and investigated. Machinekeys were then targeted, checked for proper ACLs, etc. and after nothing worked, the conclusion was metabase corruption, and IIS should be uninstalled and reinstalled. This solved the issue, but for all the wrong (and unknown) reasons.

This had occurred, however, on 3-4 other occasions in the customer's environment, so it was time to find out why. Like clockwork, the problem occurred again, so thanks to a friend in Exchange (you know who you are) asking me to give my .02 cents, I got my chance to look at it.

Epilogue (the good part)

In my experience, the reason IISADMIN fails to start in this scenario is because of a failure to decrypt secure data in the metabase, which, of course, brings us back to our old friend Machinekeys. The machine key is a file that IIS uses to decrypt secure data in the metabase. There can be many machine key files in the MachineKeys directory, but the IIS specific one starts with "c2319", which I remember thanks to the memorable quote in Monsters, Inc. (and now you will too).

There's a little known fact that the machine key file IIS uses can be stored in one of two locations. The typical one everyone seems to know is:

%ALLUSERSPROFILE%\Application Data\Microsoft\Crypto\RSA\MachineKeys

The other location that is not so well-known is:

%windir%\Profiles\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

This is almost always due to the fact that the machine had been originally upgraded from NT4. We looked in the registry under the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders

and confirmed the Common AppData value was, indeed, pointing to the C:\Windows\Profiles directory! Now we're getting somewhere!

Next we made copies of the multiple sets of machine key files, deleted the "c2319" machine key file in the \Windows\Profiles\...\MachineKeys folder and then copied the one from \Documents and Settings\...\MachineKeys into the \Windows\Profiles\...\MachineKeys folder and voilà this was a match! IISAdmin started just fine.

The keen reader should notice this means IIS WAS using the \Documents and Settings\...\MachineKeys folder initially and something changed it after the fact. This is the reason for my almost comment above. Upon further inspection, it appeared the customer had installed a piece of older software (NT4 version, perhaps?) that actually modified the registry to point to the \Windows\Profiles\...\MachineKeys directory. OUCH!

One other tidbit I noticed during the course of working on this issue is if you delete the machine key IIS uses, then attempt to start IISADMIN, another "c2319" machine key file will be created using the same filename. The problem with this is the data/private key inside will not match what was used to encrypt the metabase data initially, therefore, it's completely useless and IISADMIN will not start. This leads me to ask why it's created in the first place. If you know, I would love to hear...

Comments

  • Anonymous
    January 20, 2006
    Very Helpfull! Please keep sharing

  • Anonymous
    February 06, 2006
    Great article!

    We had similar problems (IIS Admin Service not starting after a reboot) and tried to solve it with restoring the IIS Metabase. Didn't help.

    But when reading your article about the machinekey, something came to my mind.

    A couple of days ago we overwrote the original All Users profile since the Citrix users needed a different 'default' setting for their countrysettings.

    With this, the original machinekey which IIS Admin Service uses, disappeared as well.

    But before we overwrote the original All User Profile folder, we made a backup. From backup we restored the original machinekey for IIS Admin Service.

    And voila, everything runs like clockwork again!

    Thanks for sharing your experience and knowledge!

    Kind regards,
    M.S. Wolf
    System Administrator (and learning a lot from mishaps like mentioned above...)

  • Anonymous
    March 14, 2006
    Bonza advice - and much beeter than the MS technet info.

  • Anonymous
    April 27, 2006
    good
    It solves my problem
    Thank you!!!!!!

  • Anonymous
    July 27, 2006
    Awesome!! This solved an issue we experienced in our CITRIX environment. We went back to tape to get the machine file.

  • Anonymous
    August 20, 2006
    Great Artical !

    But In my case we end up in re-installing the IIS because unfortunatelt for in our Backup configuration "C:Documents and Settings" was under exception list and never get backed up.

    Lesson learned:
    If you have IIS Installed, make sure your  Tape backup application includes "C:Documents and Settings" or Atleast "C:Documents and SettingsAll UsersApplication DataMicrosoftCryptoRSAMachineKeys"

  • Anonymous
    August 21, 2006
    I ran into the same problem on Win2k3 R2 x64. I didn't try to use it after setting up the machine but based on the logs it wasn't working initially. This server was created from an image (the image is created by another team).

    I don't have the windowsprofile directory mentioned and the reg entry points to the All Users folder as it should, so at this point I'm going to uninstall/reinstall and see if that works.  Based on the article I don't see any other option.

  • Anonymous
    November 30, 2006
    Great article! Some very useful info..Thanks

  • Anonymous
    December 04, 2006
    Thank you! This blog just saved my life!! Thank you for these insights.

  • Anonymous
    January 21, 2007
    Hello! I am Billy Johnson Nice design. Enjoy! Good site! OK. 0n79p7k .

  • Anonymous
    February 07, 2007
    I don't have a  windowsprofile dir.  I tried to reinstall and still no go.  Any suggestions?

  • Anonymous
    February 11, 2007
    Hi all , very nice site! Thank You !

  • Anonymous
    February 20, 2007
    Thanks a lot, a great article!!!!! I can resolve the problem.

  • Anonymous
    April 04, 2007
    JEPP very fine stuff and thanks a lot :)

  • Anonymous
    July 11, 2007
    The comment has been removed

  • Anonymous
    July 25, 2007
    Thank you for this article. This saved me from reinstalling my server. And always remember: It is a rather bad idea to move the "All Users" profile folder to another drive without moving the files (like the RSA keys) as well.

  • Anonymous
    July 27, 2007
    This article is a life saver. Keep up the good work. I initially thought I had to reinstall IIS but after reading this post and following it to the T I did not have to re-install IIS. Thanks a million.

  • Anonymous
    August 07, 2007
    (sigh)  No, uninstalling GoToMeeting wasn't the solution, and following the advice in this column wasn't either.  My machine wasn't updated from an earlier version (fresh into XP Pro), and "Common AppData" is pointing to the correct folder in the appropriate %ALLUSERSPROFILE% directory.   Although I do have a C:WINDOWSProfilesAll Users directory, all that is in it is an empty C:WINDOWSProfilesAll UsersAdobeWebbuy folder (maybe a clue to something, but I'm not seeing the connection).  I also have only one "c2319" machine key file on my computer (in the correct directory). I am very glad this information has helped a lot of people.  I, for one, will have to look further. Thank you for your efforts.

  • Anonymous
    August 14, 2007
    Okay... First thank you for the posting as it put me on the right path and saved me a lot of time. I had a similar issue with the IISAdmin Service not starting. I noticed that I had two c2319 keys in my folder. The dates were so that one was before our issue and one was after. I renamed the new (not working) key and tried to start the service. It didn’t work and when it created the new key, it came back as the wrong one (although the same wrong one as before. The keys are set in two groups of numbers with an “_” separating them. The first set is not important here, but the second set of numbers (after the _) is the key represented in the registry as MachineGuid and can be found at HKLMsoftwareMicrosoftCryptography. I changed that to match the old key and everything started working as normal. Still testing to see what the fix might have broken, but so far so good.

  • Anonymous
    September 05, 2007
    Thank you, Kane. Your last post is the one it help us. Great article!

  • Anonymous
    January 05, 2008
    PingBack from http://birthdays.247blogging.info/?p=3560

  • Anonymous
    February 27, 2008
    THANKS YOU!!!!!!!!!!!!!!!!!!!!!!!!!!!

  • Anonymous
    March 27, 2008
    PingBack from http://www.dscoduc.com/post/2008/03/Cardspace-for-BlogEngineNet.aspx

  • Anonymous
    May 14, 2008
    Thanks for the tip! Not sure why but on mine copying the c2319.. file from C:Documents and SettingsAll UsersApplication DataMicrosoftHTML HelpCryptoRSAMachineKeys  to   C:Documents and SettingsAll UsersApplication DataMicrosoftCryptoRSAMachineKeys allowd IIS admin to start.

  • Anonymous
    May 16, 2008
    PingBack from http://www.puzzlebird.com/wp/technical-hands-on/restore-iis-when-iis-admin-fails-to-start/

  • Anonymous
    July 15, 2008
    My research of why IIS 6 could not start brought me to this thread, but restoring keys did not work for me. My problem turned out to be a corrupt metabase. Restoring MetaBase.xml fixed my issue. To restore MetaBase.xml, restore system state from your backup to a temp folder. Now copy the restored MetaBase.xml from your temp folder to %Systemroot%System32inetsrv

  • Anonymous
    August 24, 2008
    I have had an older (manual)backup of the IIS-Metabase-configurationfiles in %windir%system32inetsrvmetaback If I tried to make a resore (after reinstalled IIS) it didn't work. (signature-failure / "Falsche Signatur"), Now we know, it's because of the new machinekey-file. My Solution was : net stop iisadmin /y net stop W3Svc /y transfer the following XML-Tags form backuck.MDO into ..inetsrvmetabase.xml :

  • WebSvcExtRestrictionList=...
  • all <IIsWebServer .. (webpages) entries until tag <IIsApplicationPools AND NOW : change all entries UNCPassword="... and AdminACL=".. into the same entry as AdminACL=", which you will find at the top of file in tag <IIS_ROOT. So you use the NEW metabase.xml, have the old entries for applications and websites in it, but simply changed their key-entries into the new key. After finishing save metabase.xml and start iis : net start iisadmin net start W3Svc For me, this was the solution for having NO Back-MachineKeyfile c2319, but had a back-config-file.

  • Anonymous
    January 22, 2009
    I am encountering this error on Windows XP SP3 after sysprep with IIS 5.1 WWW and FTP installed.  I am using the SP3 version of sysprep, so I don;t have any incompatibility there.  Any idea how to prevent this from happening during/after sysprep?

  • Anonymous
    February 19, 2009
    Kane Martin? you are the BEST!

  • Anonymous
    March 17, 2009
    Your solution worked!  Thanks so much for the explanation and saving me all the troubleshooting and headaches just to figure this one out!   Thanks again for sharing!

  • Anonymous
    June 13, 2009
    話題の小向美奈子ストリップを隠し撮り！入念なボディチェックをすり抜けて超小型カメラで撮影した神動画がアップ中！期間限定配信の衝撃的映像を見逃すな

  • Anonymous
    July 26, 2009
    Thank You !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  • Anonymous
    August 15, 2009
    Thanks for this article, and a big thanks to Kane Martin. Had the same problem in my environment. For some reason, mid-week, the 'c2319' MachineKey file was replaced by a new one with a different MachineGuid. Swapped them around and IIS (and most importantly, SMTP) started right back up. Thanks again!

  • Anonymous
    October 05, 2010
    Great! it works for me. Thank you! And I hate Windows...

  • Anonymous
    March 12, 2014
    I found the answer in this article, with a small twist. On our Win 2003 machine there was a C23 key in C:Documents and SettingsAll UsersApplication DataMicrosoftCryptoRSAMachineKeys AND C:WINDOWSDocuments and SettingsAll UsersApplication DataMicrosoftCryptoRSAMachineKeys. I copied the key from C:Documents... to C:WINDOWSDocument... and was able to start IIS just fine.