The Pitfalls of Using Multiple Patch Management Services on Windows 10

  • Article

Hello again fellow Windows 10 users!

 

I'd like to dedicate this post to discussing a reoccurring problem that I am seeing crop up in more and more customer environments, where multiple patching services (such as MS System Center Configuration Manager and IBM Bigfix) are being utilized concurrently to facilitate the patching of Windows 10 devices. The use of two different patching solutions within the same environment can lead to a very unique situation being encountered where Windows 10 will not initialize the OS properly, leading to users getting stranded in the Windows Recovery Environment.

 

The back story.

 

To really get to the bottom of what's happening, you need to be familiar with the patching methodology that Microsoft is utilizing to ensure that Windows 10 stays as up to date as possible. When Windows 10 became generally available to the public, a new patching paradigm was adopted to keep the code that Windows 10 runs off of as up to date and secure as possible for all of our customers. Monthly Cumulative Updates (referred to as CUs from this point on) became the new patching mechanism responsible for delivering all the necessary code changes to the Operating System. Anyone who has ever worked with CUs knows that there is one predominant problem when it comes to CUs... mainly that they are, well, cumulative! Each months update gets progressively larger because of ever increasing amounts of updated code that get added to the updates.

To combat the growth problem, shortly after the release of Windows 10 version 1607 (Anniversary Update), Microsoft introduced a new Windows Update package type effectively dubbed the Delta update. Like the name implies, it only contains code that has changed since the last full CU that was released, which effectively (and dramatically) reduces the overall size of the updates that get pushed out to managed systems. Cool right?

Unfortunately, these two patch types have introduced some peculiar issues into customer environments... especially environments that rely on two different patch deployment solutions.

 

The problem.  

 

The main problem that arises from utilizing multiple patch deployment services is that it becomes possible to potentially deploy both types of updates concurrently. When it comes to the issue mentioned in the first paragraph of this article, this is in fact exactly what's happening, which is typically what leads to Windows 10 devices ending up a non-bootable condition. The problem state occurs when both a full CU and Delta CU get deployed during the same reboot cycle from both patch management solutions.

Picture this scenario if you will. Your organization utilizes IBM Bigfix to catch and deploy the full CU out to all Windows 10 machines that need it, and MS System Center Configuration Manager to deploy the Delta updates. A Windows 10 machine that is a month or so behind on it's updates gets booted up and analyzed by both services. BigFix sees that the device needs the latest deployed monthly CU, and pushes the update down to the system. No restart occurs due to policies that are in place that prevent this kind of action from occurring during work hours. The Configuration Manager Agent kicks in directly afterwards, and detects that the machine needs the latest Delta update that has been published. This update gets pushed down to the system and installed. No restart occurs due to the same policies mentioned above. When the machine does finally reboot, we immediately encounter the non-bootable state mentioned above. So why is this? Lets dig into the issue a bit.

If we take a look at the stop error code that gets generated by the system during startup, we will see that a 0x7B INACCESSIBLE_BOOT_DEVICE stop error code gets thrown. This error is a direct result of the oddball boot state that the system gets into when both a full CU and Delta update get pushed to, and installed by a system during the same reboot cycle.

 

Recovery.

 

So great. You've found yourself stuck in this state. Now what? Fortunately, it's fairly easy to recover Windows 10 devices from this non-bootable condition. Unfortunately, the use of Bitlocker or other third-party encryption service like McAfee or CheckPoint tend to slightly complicate the recovery process. You will quickly find that you need to decrypt or suspend encryption on these system prior to being able to perform the recommended fix action below.

For Bitlocker users, suspending encryption is a fairly straight forward and easy process. Use the following commands below to suspend encryption so that the repair operation can be performed:

manage-bde.exe -protectors -get c:
manage-bde.exe -unlock c: -recoverypassword "xxxxxx"
manage-bde.exe -protectors -disable c:

In the case of other third party encryption solutions, I've seen the decryption process take anywhere from a couple of hours, to three or four days... depending on the encryption service that's being utilized. If a customer has a wealth of data that has not been backed up on their system, it will likely be a very worthwhile action for you to perform.

The reason the device needs to be either decrypted or unlocked is due to the fact that we will need to utilize some recovery tools from the nifty Windows Recovery Environment that will need access to the volume that Windows is installed on... namely, the Command Prompt, which can be found under the Advanced Options section of the Recovery Environment:

 

Once you have gotten into the Command Prompt, you're going to want to run the following command:

  • dism.exe -image:C:\ /get-packages > X:\packages.txt

*Note - The -image option should point to the volume where the Windows directory is located.

  • Open the packages text file with notepad by typing: notepad x:\packages.txt

 

At this point you will want to scroll all the way to the bottom of the text file, and look for any packages that have a state status of Install Pending. Copy the package names, and use the following command to remove the 'Install Pending' packages:

 

  • dism.exe /image:C:\ /remove-package /packagename:Package_for_RollupFix~31bf3856ad364e35~amd64~~14393.953.1.2

*Note -  The above package name was included only as an example. Please ensure you use the package name from the packages.txt file above.

 

Once the DISM tool has finished removing the package, your system should be good to go. Once it's reboots, Windows 10 will continue to uninstall any packages previously scheduled for removal, at which point the OS should be able to initialize itself properly.

 

The follow up.

 

To prevent the non-bootable issue from reoccurring in the future, it is highly recommended that customers configure their patching infrastructure so that the Windows 10 full CUs and Delta updates are NOT installed at the same time, during the same reboot cycle.

 

And there you have it folks! Go forth and perform good updating deeds! Until next time.

Comments

  • Anonymous
    October 24, 2017
    In our case, MS just introduced Delta updates and they were not filtered out because it was something new so they were deployed along with the cumulative update. Other MS patches check to see if they are applicable, it seems like poor coding practice that the Delta and Cumulative wouldn't check for the existence of the other before installing.
    • Anonymous
      November 09, 2017
      The mechanism that is involved in the deployment of cumulative and delta updates to Windows 10 machines is definitely not ideal! Our product group is very much aware of this issue, and is actively working on remedying the circumstances that can lead to these kinds of issues at the highest levels. I've been told that there should be a code change soon that will prevent this issue from occurring in the future.
      • Anonymous
        November 15, 2017
        Thank you!