Why one critical production server rebooted at 3:20am this morning?

  • Article

Recently I have done a Configuration Manager 2012 R2 engagement to assist customer configuring Software Update feature. The customer wants to use Configuration Manager 2012 R2 to deploy security updates to all production servers. Like most customers originally using WSUS for security updates, this customer wants to have better controls and detailed reports in Configuration Manger 2012 when deploying security updates. One request the customer has is to keep using the existing WSUS server.

 

The Software Update Point is successfully configured using the existing WSUS server.  But software update scan is failed on all servers. Report "Scan 3 - Clients of a collection reporting a specific state (secondary) " in "Software Updates - D Scan" category shows the reason. The screenshot below is from my testing environment. Hundreds of servers reported the same error “Group policy conflict” in the customer’s environment:

It is a known issue and we need reconfigure the Active Directory Group Policy, "or move client computers to an organizational unit (OU) that does not have this Group Policy setting applied" ("Planning for Software Updates in Configuration Manager - Group Policy Settings for Software Updates"). The customer decides to unlink the WSUS domain Group Policy from the production server OU in Group Policy Management console.

The next day when I arrived customer’s site, the customer told me one critical production server was rebooted at about 3:20am this morning. His boss wanted a full investigation on the cause of the reboot. He asked my assistance to check the production server.  

 

The investigation is a short one. First we found the server was rebooted due to security updates installation, because System Event Log captured Event ID 22 from “Automatic Updates”. It says: “Restart Required: To complete the installation of the following updates, the computer will be restarted within 15 minutes”   

 

Then we checked Control Panel\System and Security\Windows Update\Change settings on the production server. The setting is “Install updates automatically (recommended) ”, with the installation time set to 3:00 AM.

Other servers are not configured with “Install updates automatically (recommended) ” setting. It seems the automatic update setting on this critical production server was configured incorrectly during the Operating System setup.

 

The content in C:\Windows\WindowsUpdate.log and WSUS Server “Automatic Approvals” options tells the whole story.  

  • The production server receives a local group policy to configure the location of WSUS server shortly after Configuration Manager Agent installation. The local group policy only changes the location of the WSUS server using FQDN name.

        

  • The WSUS domain policy is configured to use WSUS server’s NETBIOS name. The automatic updating is configured to “2 – Notify for download and notify for install” in the domain group policy. There will be no automatic update installation with this domain group policy enabled.

       

  • When the WSUS domain policy is unlinked, the production server reverts to its default automatic updates setting, which is “Install updates automatically (recommended) ” at 3:00am. The production server will connect to WSUS server using its FQDN name, as configured by local group policy after Configuration Manager Agent installation.
  • The Software Update Point configuration DOES NOT change “Automatic Approvals” rule when the same WSUS server is used. The “Default Automatic Approval Rule” is still enabled. It means any production server with “Install updates automatically (recommended) ” automatic updates setting, and configured to use this WSUS server, will download and install updates automatically.

           

 

Here is the lesson learnt:

===================

The following tasks should be performed before unlinking any WSUS domain group policy:

  • Create a new domain group policy to disable local Windows Update setting before unlinking the WSUS domain group policy.

       

  • Create a compliance baseline to verify the new domain group policy has been successfully applied to all servers with Configuration Manager Agent.

     

      

 

From IT Operations perspective, operations such as unlink domain group policy should be done on a test OU with a small number of production servers, before apply the change to all production servers. It will ascertain any unforeseen effects.

Comments

  • Anonymous
    January 01, 2003
    All images are uploaded.
  • Anonymous
    September 30, 2014
    Missing some images....
  • Anonymous
    October 01, 2014
    Thanks
    very interesting