How to manage Pending Reboots of your AD-Integrated servers using Orchestrator

Introduction

Windows Servers can request for a reboot when:

• A new patch was applied
• An administrator installed or removed a Windows Server role / feature
• A change was made on the system that requires a reboot

Rebooting a server immediately when it asks for it usually ends up with frustration from the customers as it might cause downtime or loss of data if it is not properly managed. In the other hand, if your environment has many servers, it would be very costly to check the servers and do the reboot manually.  This Wiki article shows how to use Microsoft System Center Orchestrator to efficiently and automatically manage pending reboots for your AD-integrated servers.

How to use Orchestrator to manage pending reboots for AD-integrated servers

To use Orchestrator to manage pending reboots for your AD-Integrated servers, all you need are:

• An SQL table that has the following columns:
  • Hostname: This column is used to store the computer name
  • rebootdelay: This column is used to store the number of seconds to wait before checking that a rebooted server is online or not
  • rebootorder: This column is used to specify the reboot order of servers

• Orchestrator Runbooks to import your servers name from Active Directory to the SQL table
• Orchestrator Runbooks to process a reboot for every server having a pending reboot

Orchestrator will rely on the data stored in the SQL table to process reboots. It will check the servers and process pending reboots in the order (rebootorder) stored in the table and will wait for a specific amount of time (rebootdelay) after a server is rebooted before checking if it is back online or not. Once the server is detected by Orchestrator as online, Orchestrator will start checking the next server and reboot it if required. If a server does not require a reboot, Orchestrator will perform no action and will check the next server.

Below is an example to allow a better understanding about how Orchestrator will process pending reboots.

Let’s suppose that we have three (3) servers to check on daily basis and reboot them if they have a pending reboot.

The next table shows the Hostname, rebootdelay and rebootorder values stored in the SQL table and if the servers have a pending reboot or not.

Orchestrator will read the data from the SQL table and will order the servers by using their rebootorder. Orchestrator will then process the servers in the order below:

Orchestrator will start with Server1 and check if it has a pending reboot. It will then detect that the server is requesting a reboot and will process it. Orchestrator will wait for 300 seconds (5 minutes) before starting to check that the server is online or not. If, after five (5) minutes, the server is online; it will go to check if Server3 has a pending reboot or not. If the server is not yet online, Orchestrator will wait for the server and will check if it is online every minute.

Once the reboot of Server1 was processed, Orchestrator will go and check Server3. It will detect that Server3 has no pending reboot and will then perform no action and go to check Server2. Server2 has a pending reboot so Orchestrator will reboot it, will wait for 500 seconds before checking if it is online or not.

After the implementation of Orchestrator Runbooks, you will need to manage manually the rebootdelay and rebootorder column values. A server that has NULL value for rebootorder will not be in the scope of Orchestrator automatic reboot.

Configuration of Orchestrator

Import of AD-integrated servers list to the SQL table

You will need to have two Runbooks to import the AD-integrated servers list to the SQL table:

• A Runbook to get the list of AD-integrated servers and invoke the second Runbook
• A Runbook to store a server name in the SQL table if it does not exist

Let’s start with the second Runbook. You will need to use three (3) activities:

• Initialize Data: It will allow the Runbook to take the Host name as input

• Query Database: It will take the Host name provided by Initialize Data activity to check it already exist in SQL table or not.

• Write To Database: It will create a new entry for the Host name if it does not already exist and will set a default value for the rebootorder column (300 seconds)

The link between Query Database and Write to Database activities need to be set with the following include condition: Number of rows from Query Database equals 0. This allows adding a new row for the Host name only if it does not already exist.

Below is a screen capture of all the activities included in the second Runbook:

Once you have created the second Runbook, you can proceed with the creation of the first one. You will need to use four (4) activities:

• Monitor Data/Time: It will allow you to specify the time of when the Runbook need to start running

• Check Schedule: It allow you to specify the slot of days and hours when the Runbook can start running (By combining the settings in the activity and the ones from the previous activity, you will be able to precise the dates and times of when your runbook can start running)

• Execute PS script: This activity will be available if you register and deploy Orchestrator Integration Pack for Powershell Script Execution (http://blogs.technet.com/b/orchestrator/archive/2013/10/02/the-orchestrator-integration-pack-for-powershell-script-execution-1-2-is-now-available.aspx). It will allow you to get the list of AD-Integrated servers by running an LDAP query (You need to specify that this activity will run against a server that has Powershell modules for AD installed). The Powershell command to run is foreach ($computer in (Get-ADComputer -LDAPFilter "(operatingsystem=Windows Server*)")) {$computer.name}

• Invoke Runbook: It will allow you to run the second Runbook and to provide the Host name for every AD-integrated server as input.

Below is a screen capture of all the activities included in the first Runbook:

Processing of reboots for servers having a pending reboot

You will need to have two Runbooks to process reboots for servers having a pending reboot:

• A Runbook to get the list of servers to process their reboots and invoke the second Runbook
• A Runbook to check if a server has a pending reboot or not. If there is a pending reboot, the Runbook will initiate a server reboot.

Let’s start with the second Runbook. You will need to use seven (7) activities:

• Initialize Data: It will allow the Runbook to take the Host name as input

• Query Database: It will take the Host name provided by Initialize Data activity to get the rebootdelay value

• Get Computer/IP Status (1): It will allow checking if the server status. If it is responding then the next activity can run

• Execute PS Script: This activity will allow checking if a server has a pending reboot or not. The check is done by querying three (3) registry entries using the following Powershell command: (Test-Path "hklm:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\RebootPending") -OR (Test-Path "hklm:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired") -OR (Test-Path "hklm:\SYSTEM\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations"). It the result is True then the server has a pending reboot.

The link between Get Computer/IP Status (1) and Execute PS Script activities need to be set with the following include condition: Get Computer/IP Status (1) returns success. This means that the checks are initiated only if the server is reachable.

• Restart System: This activity initiates a system reboot when the server has a pending reboot.

The link between Execute PS Script and Restart System activities need to be set with the following include condition: PS Execution 01 Results from Execute PS Script equals True. This means that the reboot is initiated only if the server has a pending reboot.

• Sleep: This activity will start a sleep on the runbook to wait for the period set in rebootdelay value before starting to check if the server is online or not. You can create this activity by following this article: http://contoso.se/blog/?p=2802

• Get Computer/IP Status (2): It will allow checking if the server status. If it is responding then the the Runbook execution will be finished for the server. This activity need to be configured with looping to check if the server is online or not every one (1) minute (This is required to avoid going the next server if the server is still not online after the delay configured in rebootdelay)

Below is a screen capture of all the activities included in the second Runbook:

Once you have created the second Runbook, you can proceed with the creation of the first one. You will need to use four (4) activities:

• Monitor Data/Time: It will allow you to specify the time of when the Runbook need to start running

• Check Schedule: It allow you to specify the slot of days and hours when the Runbook can start running (By combining the settings in the activity and the ones from the previous activity, you will be able to precise the dates and times of when your runbook can start running)

• Query Database: It will identify the servers in the scope of the management of Pending Reboots by query the SQL table for the rows having a value for rebootorder and then it will order them to be processed by the second Runbook.

• Invoke Runbook: It will allow you to run the second Runbook and to provide the Host name for every server in the scope of Pending Reboots processing as input.

Below is a screen capture of all the activities included in the first Runbook:

Conclusion

This Wiki article shares a way to manage pending reboots for AD-integrated servers. The proposed method is suitable for non-clustered servers and allows minimizing the downtime and avoiding the data loss if the reboot processing Runbooks are scheduled to start out of business hours where you are sure that no activity is done that time. The provided Runbooks can be customized to meet your company requirements (Example: You process the reboots per platform, per technology ...). Once implemented, all you need to manage will be the rebootdelay and rebootorder variables for your servers by updating the SQL table.