Using PowerShell as an Azure ARM REST API Client

Article
06/28/2017

Yes, this is probably another post explaining how to use Azure ARM REST API using PowerShell, I am aware of this, but what I would like to show you is something deeper in the Azure platform that you may not have noticed or seen before. Fortunately, Microsoft provides many SDKs for almost all your favorite languages, then how the platform works at the lower level, from an API perspective at least, is almost entirely managed for you and then invisible. I am assuming here you are using Azure Resource Manager (ARM) API, Azure Service Management (ASM) API are not recommended today since obsolete and not interested by platform new features development.

Introduction and Goals

Just mentioned above that there are already several tools available on the Internet that will permit you to use ARM REST API directly and making web requests, let me list here first the most popular and my favorites:

Postman: you can download from here versions for Windows, Linux, MacOS and as a Chrome extension. Nice tool, available in free basic version and pro paid. powerful GUI platform to make your API development faster & easier, from building API requests through testing, documentation and sharing.
HTTP Master: grab your free basic or professional paid version from here. in addition to basic features, it provides almost complete coverage for development and testing of API applications and services, including HTTP tool to simulate client acitivity.
ARM Client: it is not an official Microsoft tool, instead is an OSS Project you can find on GitHub or can install via Chocolatey. Nice article from David Ebbo if you want to know more.

There are also many articles that will explain you how to use Azure ARM REST API, then why I decided to publish this post focused on PowerShell? Long story short, I recently worked on two different projects aiming the same kind of final result: write everything necessary to create and manage a series of Azure resources, using plain ARM REST API calls. In the first case, my partner wanted then to include these REST calls in their Java application. In the second case, my other partner wanted to include REST API in their deployment engines using different components written in different languages (Python, .NET, PowerShell), then requiring a common denominator to avoid re-writing code. Another important aspect for the first project was the requirement to have access to new Azure features immediately, or at least as soon as possible once something new feature is announced.

How new Azure API are published?

This is an important first detail to be aware of: practically, there is no new Azure feature if there is no REST API available to use it, and obviously you need to have API reference and documentation to use. What happen normally, when Microsoft release a new feature, is that REST API are built at the very first, and described here on GitHub:

What you will find in that repository are the REST API full details for all Azure things, decorated? with Swagger specs. If you don't know what is Swagger and want to know more, you can go here, or take my short and trivial scholastic definition: Swagger is a specification for machine-readable interface files for describing, producing, consuming, and visualizing RESTful Web services?.

Finally, what Microsoft internally does is using a tool called AUTOREST (and something more called extensions?) to automatically generate client SDKs and make available to you for consume. You can find it here on GitHub:

Swagger (OpenAPI) Specification code generator featuring C# and Razor templates.

Supports C#, Java, Node.js, TypeScript, Python and Ruby.

Let me emphasize this: official ARM REST API documentation should be updated to reflect the change soon, but it is worth nothing that this would require some time since human intervention is necessary to write, review and publish content:

Azure Resource Manager

/en-us/rest/api/resources

Then, if you want to know as quickly as possible about how to use a new Azure feature, you should go to GitHub and look at REST API specs. Documentation will follow soon.

NOTE: At the time of writing, there are no Swagger specs for Azure Storage API.

If you are a Visual Studio 2017 user, and interested in building Web API using Swagger, you may want to review the article below:

Visual Studio 2017 and Swagger: Building and Documenting Web APIs

https://www.simple-talk.com/dotnet/net-development/visual-studio-2017-swagger-building-documenting-web-apis

PowerShell Pre-Requisites

Now that you are a bit familiar with Azure REST API and how they are published, and heard why I used PowerShell to consume these APIs, let me start showing you a practical example on how to build REST API calls, invoke them and manage the results. But before going into details, you need something to prepare at the beginning of your PowerShell script, here is the list of logical steps:

Initialize your script with common variables values you will use all over the script. This will eventually include Subscriptions details (ID and Name), Azure region, create common objects like a default Resource Group, you Azure Active Directory (AAD) tenant.
Authenticate to your Azure subscription using Azure Active Directory (AAD) account: this step will use interactive authentication (Login-AzureRmAccount?) just to create Application? and Service Principal? objects in AAD. This is required only one time: once created the necessary objects for application logon (see later), your application/script will be able to logon to AAD programmatically without any user intervention.
Create an Application object in AAD (New-AzureRmADApplication?). Azure offer developers the possibility to build applications that can be integrated with Azure Active Directory (AAD) to provide secure sign in and authorization for their services. To integrate an application or service with AAD, a developer must first register the details about their application with Azure AD. This can be done using the Azure Portal or using PowerShell script as in my code sample below. The important piece of information you will need to obtain from this step is ApplicationID? that you will use as a surrogate User Name? for creating logon credentials.

Use portal to create an Azure Active Directory application and service principal that can access resources

/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal

Once created the Application? object in AAD, and obtained its ApplicationID?, you need to create a Service Principal object using ? New-AzureRmADServicePrincipal? cmdlet, associated to the ApplicationID?. Why you need this object? It is important to understand the difference between Application? and Service Principal?. You can read more at the link below, but in short, you can consider the application object as the global representation of your application for use across all tenants, and the service principal as the local representation for use in a specific tenant.

Application and service principal objects in Azure Active Directory (Azure AD)

/en-us/azure/active-directory/develop/active-directory-application-objects

With the Service Principal object in your hands, you can use now use to assign permissions in the current AAD tenant. In the example below you can see both scopes supported: at the entire Azure subscription level, or scoped to a particular Resource Group (recommended). You can use both default built-in AAD roles or create new custom ones, depending on your requirements.
Next step consists in creating a new Key? in AAD, associated to your just created Application?, and then store for later use in your script. You can read here instructions to do this using the Azure Portal (key chars scrambled).

Using nice PowerShell mechanism, as you can see in the example below, you can now create encrypted credential set (user name and password) to pass to your subsequent cmdlets. Be aware that "ApplicationID" is your username and must be used in the form of "username@domain-name" ?, where the domain-name looks like " <something>.onmicrosoft.com". Last piece of information you need to retrieve is AAD Tenant Name for your subscription, you can easily retrieve using last PowerShell script line shown below.
From now and going forward, you can logon using just created Application identity and secret, you don't need interactive logon anymore. After execution of the new instance of Login-AzureRmAccount? with application parameters, you will be now authenticated as an application in the script.

We are now at the final preparation step, probably the most important one, that is getting a Bearer? authentication token from AAD endpoint (https://login.windows.net/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/token) ? using Oauth2 (key chars scrambled):

As you can see above, I used "Invoke-RestMethod" PowerShell cmdlet to send HTTPS requests to Representational State Transfer (REST) web services, in this case a PUT? to Azure AAD Oauth2 passing ApplicationID and Key, for the ARM resource endpoint (https://management.core.windows.net) . Token as a form similar to the one below (text is scrambled). Be aware that the default life time for the token is 60 minutes (3600 seconds), after that you will need to generate another request and acquire a new one.

Azure recently introduced the possibility to change the default lifetime of an AAD token feature is still in preview and I did not personally use it yet, but you may want to read the article below:

Configurable token lifetimes in Azure Active Directory (Public Preview)

/en-us/azure/active-directory/active-directory-configurable-token-lifetimes

Building and Executing REST calls

Now we have a valid AAD token and we can finally do something interesting with Azure ARM REST API. It is important to highlight that every REST call must include authentication token in the header, but I am sure will be clear once you will see below the first trivial example, that is how to retrieve details about your Azure subscription with a simple GET verb:

Please be aware that the result is not JSON formatted by default, you need to explicitly convert, then you will see a similar output as the one in the picture below. It is also worth mentioning that specifying the API version is mandatory. Finally, you can see there is no Body part? defined in this simple GET request.

The sample above is pretty simple, at the end is a GET request getting synchronous answer. But what happen if I want to write (PUT for example) something? Look at the example below used to create a storage account:

NOTE: in the code snippet above, I started using Invoke-WebRequest? instead of Invoke-RestMethod? because I had problems in correctly parsing and processing the response header, I normally prefer the former.

In the response content above there are several interesting things: first, the status code returned in StatusCode is (202), that is Accepted? (StatusDescription?) in HTTP response code vocabulary. It is indicating that the request you submitted is a long running? asynchronous operation, then the system returned your code a result, but the backend operation is still running in the Azure ARM infrastructure. For synchronous operations like typical GET methods to read objects and resources, you normally receive 200 (OK) .

Microsoft REST API Guidelines

https://github.com/Microsoft/api-guidelines/blob/master/Guidelines.md

According the excellent documentation above, "Retry-After" HTTP header indicating the minimum number of seconds that clients SHOULD wait before attempting the operation again. Why this is important? Azure REST API infrastructure tends to protect itself from being overloaded, then this value is returned in all async responses. But, if you do not want to obey to this hint, there is another safe limit that Azure enforces, you can guess looking to x-ms-ratelimit-remaining-subscription-writes? element. For each subscription and tenant, Azure Resource Manager limits read requests to 15,000 per hour and write requests to 1,200 per hour. When you reach the limit, you receive the HTTP status code (429), that is "Too many requests". You can see all the limits in the article below along with additional details.

Throttling Resource Manager requests

/en-us/azure/azure-resource-manager/resource-manager-request-limits

Now, if you want to check the status of your running async operation, you need to execute some code like the example below:

And here is the content of the variable $URIAsyncCheck? (single line scrambled):

General rule to track async operation, for Azure ARM REST API, requires you to check a specific dynamically crafted URL for the status: this URL is generated depending on the ARM Resource Provider (RP) you previously sent the async request to, in this case the Microsoft.Storage? RP, and the OperationID of that request. You can see these elements highlighted in red in the above text. Unfortunately, there is no single place you can retrieve that OperationID: as you can see in the code snippet above, you have first to check for the existence of value for Location? attribute in the response header, then if nothing is in there, you need to check Azure-AsyncOperation?. Â With this knowledge in mind, let us check the long running operation using the code snippet below:

This is the output you should see:

As you can see, this is a long running async operation (Response Code = 202 Accepted). You can also note that until the Response Code will not change to 200 - OK, there is no body content returned (Content Length = 0). You should then wait in a loop and check periodically for this condition, once happened, you should check for provisioningState? attribute value returned: if, and only if, the operation is completed, then possible values are "Succeeded", "Failed" or "Canceled". Different values could be returned, depending on the Resource Provider you may see different values, but this indicates that the request is not completed yet. Some operations that you could think as being asynchronous, in reality they are not, as in the example below related to the deletion of a storage account (Sample[5] in the sample code):

Here the only difference is the Method = 'Delete' ? part, and result HTTP code will be 200 - OK (if the storage account exists and you have permissions). No attribute "Location" or "Azure-AsyncOperation" will be generated to check for async operation. You can see details for this particular REST API at the location below:

Storage Accounts ARM REST API

/en-us/rest/api/storagerp/storageaccounts

Going through the examples, I refined and sometimes modified the logic to track async long-running operations, a sample (extracted from the GitHub code) is below:

Building REST API request Header and Body can become difficult for very large objects as Virtual Machine. You can see an example below, please note that only the minimum required parameters are used, there are many more optional that I did not include. If you want to work using this approach, you need to keep in your hands the Azure ARM REST API reference pages below, and the -Debug? switch in PowerShell, you can read more details on the latter in the next section.

Azure REST API Reference

/en-us/rest/api

Code Samples on GitHub

Here on GitHub, you can find the code and how to deploy, end- to-end, an Azure Virtual Machine, including NSG, VNET and IPs, in a new Resource Group all using REST API direct call in PowerShell. Code for Application and Service Principal creations in Azure Active Directory is also provided. Here is the sample list divided in PowerShell regions?:

Purpose of this work is to show how to work directly with Azure ARM REST API using PowerShell as an ARM client. Code is highly de-normalized to make each section pretty autonomous and self-contained to facilitate reuse. At the end, what you should be able to obtain is the following list of resources, all of them built using plain REST API built and executed in PowerShell.

PowerShell Hidden Gems

I have been surprised many times why most people don't know the power of Azure cmdlets when adding -Debug? switch at the end of the command string. Azure PowerShell module is a wrapper built on top of Azure ARM (in this case) REST API, but if you add it?, you will see the raw underlying REST API request built for you, and the correspondent result:

Get-AzureRmStorageAccount -ResourceGroupName $rgname -Debug

Then, if you want to work at the pure REST API level, but want your life a bit easier, you could use higher-level PowerShell cmdlets and then reverse-engineer them using the -Debug? switch. Another very useful feature you can use in PowerShell is Stop Watch?. Starting Sample[6] in my GitHub code example, I used it extensively to monitor execution time of my REST API: below you can find a very simple general structure of its usage.

Java Sample

If you want a Java version of something similar I have shown you here, I would encourage you to look at Silvano Coriani (from AzureCAT) sample work on GitHub here, very nice if this is your favorite development language.

https://github.com/scoriani/ServiceBuilder/blob/master/src/com/microsoft/azure/provisioningengine/ProvisioningEngine.java