Azure Image Management Automation

Hello, in this blog post, I will talk about the Azure Image Management Automation which is a set Azure Automation Runbooks that allows us to alleviate some of the overhead work that comes with the adoption of custom “golden” images built on-premises and used in Azure.

This solution is a team effort between John Knightly (which is also a PFE) and me, where he could propose an initial solution to his customer and after some discussions between us we could design, develop  and test this solution.

Overview

To give a little perspective of how does this solution may help you, lets think about the manual process of having to build a custom golden image (steps 1-3 will exist anyways):

1. On Hyper-v or whatever hypervisor solution you use, install the OS from an ISO
2. Perform your customizations
3. Run SYSPREP (for Windows) or waagent -deprovision+user (for Linux)
4. Upload the VHD to a Storage Account by using Azure Storage Explorer, AzCopy, Add-AzureRmVHD, etc.
5. Create a managed Image of this VHD in order to be able to easily deploy VMs from that golden image

These steps are actually very simple but that simplicity stops when you need to make that image available to a second region, in this above example I uploaded the VHD in West US region and into a single subscription, now, what if you need to start deploying VMs from the same golden image throughout one more region? Simple, copy the VHD from the Storage Account in West US to West US 2 and then create a new managed image in that region. Why is that? Because managed images can be only created from VHDs located in the same region. Now, let’s add more, what if you have a DEV, QA and PROD subscriptions? Well, multiply the two steps by 3, giving you 6 operations. Expanding a little bit more, add a Windows and Linux golden image. This math will go to 12 operations you need to manually execute and monitor. Do that each month if you have a monthly golden image release cycle. Already too much work for a small shop correct?

To close this thought, imagine you’re a big global company with lots of business units and each one with its own requirements on Azure, leading you to have 300 subscriptions, be present in at least 10 Azure regions and need to maintain at least two golden images, one for Windows and one for Linux? Well, that will result in 6,000 manual operations a month. This would result in lots o man/hours of work and be error prone.

This solution helps you to simplify and automate this process, what you need to do is to continue producing your golden images and just use a single script to upload the VHD and wait the images being created in each subscription and region.

It uses the following Azure components:

1. Azure Active Directory
2. Azure Storage Accounts
3. Azure Storage Tables
4. Azure Storage Queues
5. Azure Automation

We are also relying on two PowerShell modules I built in order to interact with Storage Tables and Storage Queues. It also depends on a module created specifically for this solution to avoid code duplication as much as possible.

This diagram summarizes the process:

The following diagram shows how the process works under the hood, from uploading the VHD to producing the managed Image.

Before I continue to more details, let me define some terms:

• Tier 0 Storage Account – Storage account that contains the configuration table, the queues for VHD copy process and Managed Image creation process. Finally it is the initial point where the VHD is uploaded first.
• Tier 1 Blobs – Multiple copies of the original VHD are made inside of the Tier 0 Storage Account to allow multiple parallel copies (this is defined during setup).
• Tier 0 Subscription – Subscription where all automation components are created.
• Tier 2 Subscriptions – Subscriptions that holds tier 2 Storage Accounts and will get the Managed Image created.
• Tier 2 Storage Accounts – Final destination of the VHDs, they can be in same or different subscription, different regions, etc.

Step by step explanation:

1. After building your VHD, use UploadVHD.ps1 script to upload it to a tier 0 storage account
2. UploadVHD.ps1 script will upload the VHD to Tier 0 Storage Account and will trigger Start-ImageManagementTier1Distribution runbook
3. Start-ImageManagementTier1Distribution will create multiple copies of the VHD in order to allow parallel copies (when using Start-AzureStorageBlobCopy, the source blob is locked until the copy is in progress, meaning that no other copy operation can start)
4. After all copies are done, a message is placed in the copy-process-queue stating that there is a VHD to distribute to Tier 2 Storage Accounts (with other information that will be consumed later)
5. Runbook Start-ImageManagementTier2Distribution is scheduled to run every hour (minimum value) and check the copy-process-queue, if there is a message there, it will trigger one runbook called Start-ImageManagementVhdCopyTier2 that resides in a different Automation Account per destination Tier 2 Storage Account. Note that this is in another Automation Account due to the 200 runbook jobs running at the same time, for a complete list of limitations, please refer to https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#automation-limits)
6. After each individual runbooks completes its execution, a message per copied VHD is placed in the image-creation-process-queue
7. Another scheduled runbook called Start-ImageManagementImageCreation will monitor that queue (hourly again) and trigger one New-ImageManagementImage runbook (located in another Azure Automation Account due to same reasons already explained, it also can be in multiple Automation Accounts, all depends on how you did the initial setup) per VHD and perform the image creation.
8. The end result is that the managed images will start showing up at the locations you defined.

Setup Guide

To quick install the solution, please refer to the Setup Guide.

For operations and troubleshooting, please refer to the Operations Guide.

Details on SetupInfo.json file

In order to provide an easy to use solution, I provided a setup script that will automate the installation process for you. But just before we go into this process, we need to review the SetupInfo.json as follows:

This JSON file contains the information needed by the setup script to perform its installation, it is basically split into four main sections as follows:

• general
• requiredModulesToInstall
• storage
• automationAccount

Following items describes in detail all sections and what is required to be changed in order to set this up in your own environment.

General section

This section mainly requires the Azure Active Directory tenant name and defines the queue names.

Section example

RequiredModulesToInstall section

This sections needs to remain unchanged unless you want to have additional modules installed in your Automation Account as soon as it gets created, This is consumed by the setup script to download the required module from PowerShell Gallery and make it available in the Tier 0 storage account for later Azure Automation Account deploment.

This is a simple array with the module names that you want to install.

Section example

Storage section

This section defines the Tier 0 Storage Account information, number of Tier 1 blob copies and other important information, it also defines each Tier 2 Storage Account which is each individual Storage Accounts that will receive a copy of the VHD, can be in the same or different region/subscription.

This section is split between tier0StorageAccount and tier2StorageAccount as follows:

tier0StorageAccount subsection

Defines information about Tier 0 Storage Account. This SA is the one that contains the configuration table, queues and the blobs to be copied.

tier2StorageAccounts subsection

This is an array that defines information about each Tier 2 Storage Account. This SA is the one that will receive the final VHD blob and will be used by the managed image creation process. Initially we asked to provide these values inside the SetupInfo.json file, now there is a PowerShell script (Scripts\GenerateTier2StorageJson.ps1) that fills this section based on a text file with the list of subscriptions or single subscription provided from the command line, described later in this document.

You will have as many Tier 2 Storage Accounts listed in this section as you need, they need to be added as an extra element of this array. Notice that if in the future you need to expand your solution to have more/new Tier 2 Storage Accounts, make sure you use the same JSON file with the additions and run the setup script again. The script will create the new storage accounts or any other extra items you place here, like new number of Automation Accounts as you will see further in this article.

Section example

You will notice in the sample section that we used a different notation for some of the values, We are adding the ^ sign in the beginning of the string, which means that the setup script will execute a PowerShell expression evaluation of the content of the string. In this case what we are referencing is a value inside of the same JSON file. $config is the object we use to load the content of this file, so you can use the paths that we are showing and reference existing values throughout the file so if you have values that repeats, you don’t need to keep copying all the time, but this is complete optional and if you think that adding the actual values will be more educational for documentation purposes feel free to use the actual values. Intention here is just to offer a way to avoid repeating content.

Automation Account section

This section defines your Automation Account settings, we will have here some general settings for the Automation Accounts plus specific sections for three Automation Account types (this is solution wise and does not mean that Azure contains different types of Automation Accounts) as follows:

• Main Automation Account, this is the one that contains the Start-ImageManagementImageCreation, Start-ImageManagementTier1Distribution and Start-ImageManagementTier2Distribution runbooks.
• Copy Process Automation Account(s) , one or more automation accounts to execute the tier 2 copy process, the number of this type of Automation Accounts will depend on number of subscriptions, regions and images, as it was previously explained, a single Automation Account can execute a maximum of 200 runbook jobs (we cap at a default of 150 in this solution) so if you have, lets say 300 subscriptions, 2 images (1 Windows and 1 Linux) and need to include two regions, you will need at least 8 copy dedicated Automation Accounts in order to be as parallel as possible.
• Image Creation Process Automation(s) , same case of the previous description, this automation account is dedicated to create the Managed Images.

The formula to easily determine the number of worker automation accounts (copy and image types described above) is :

(Number of Subscriptions X Number of Images X Number of Regions ) / 150 = Value to be used on workerAutomationAccountsCount value.

This section contains the following items:

Runbooks subsection

This section defines all necessary runbooks to setup in each Automation Account type.

It is split between mainAutomationAccount, copyProcessAutomationAccount and imageCreationProcessAutomationAccount and for the sake of trying to keep this post as small as possible (which is being hard to ) I’m describing one element that is used to define a runbook in any of the sections.

Element Name	Description	Modification Required?
name	Runbook Name.	No
scriptPath	Path to the script to be imported into the PowerShell runbook. This path can be a http path as well, when you download the script directly from GitHub raw path for example.	No
scheduleName	Name of the schedule to create and assign to the runbook, the type is hourly schedule only.	No
startTimeOffset	This value is used to set the start time of the schedule, in minutes, it tells how many minutes to get started with the schedule during the schedule creation, good for staggering between tier 2 copy runbook and the image creation runbook.	No
executeBeforeMoveForward	Forces the execution of the runbook as soon as it gets imported and wait for it to finish. In our case we add the update runbook from GitHub and execute it in order to update all existing modules in the Automation Account since they are outdated at the moment of account creation, if we don’t execute it all other runbook imports fails.	No
parameters	This element define an array of key and value pairs, this defines all parameters that the runbook needs when scheduling its execution. For example:	No
requiredModules	An array of modules to import in this automation account if not imported yet. The module list item needs to come from the requiredModulesToInstall section.	No

Section example

This section is highly sensitive to what runbooks to install, order and modules so if changing it be sure to don’t change the already existing items.

That’s it for this blog post and I hope that this can help.

Regards

Paulo

Comments

• Anonymous
  August 16, 2017
  Great work Paulo, I appreciate the name drop, but this is your code! :-)
• Anonymous
  August 20, 2017
  Thanks Paulo. This will absolutely help me expand on a managed disks and image automation script I developed. Awesome work.
  • Anonymous
    August 24, 2017
    Thanks Charles, glad to know this is helping you.
• Anonymous
  August 20, 2017
  Awesome stuff Paulo. This will totally help me expand on a managed disk and image automation script I developed. Thanks man.
• Anonymous
  August 30, 2017
  I totally appreciate your work, but isn't this overcomplicating this a lot?Why not use packer to build your image on Azure? It's even now an available task on VSTS and has been around for a long while to build images on many platforms.
  • Anonymous
    September 08, 2017
    Hi David,Thanks for your comments. It depends on the scenario, this solution was built to a customer that requires their images to be built on-premises due to some security compliance, also, the solution proposed was built to scale, the initial scenario of that customer have more than 300 subscriptions using at least 12 regions each, (although not tested) this scales to any number of subscriptions and regions to cover, we do have some customers with up to 1900 subscriptions, so using a single script to upload the VHD and have it taken care for you automatically in these 1900 subscriptions seems a great deal :-) .RegardsPaulo