How to create resources at scale using the Azure CLI

Article
05/27/2024

As an Azure resource manager, you frequently have to create multiple Azure resources when configuring new environments. You might also have an Azure resource approval process that works best when Azure resources are created automatically from a script.

In this article you will learn the following:

Create multiple Azure resources from parameters received from a delimited CSV file.
Use IF..THEN statements to create dependent Azure resources.
Log script progress to a local TXT file.

This sample script has been tested in Azure Cloud Shell in both Bash and PowerShell environments, and PowerShell 7. Find the CSV and full script in Azure-samples/azure-cli-samples.

Prepare your environment

Follow these steps to prepare your environment to run the example script:

Open either the Bash or PowerShell environment in Azure Cloud Shell. For more information, see Quickstart for Bash in Azure Cloud Shell.
Download and save to a local directory the following CSV file. Replace myExistingResourceGroupName in line three with an actual resource group name.
```
resourceNo,location,createRG,exstingRgName,createVnet,vnetAddressPrefix,subnetAddressPrefixes,vmImage,publicIpSku,Adminuser
1,eastus,TRUE,,TRUE,10.0.0.0/16,10.0.0.0/24,Ubuntu2204,standard,
2,eastus2,TRUE,,FALSE,,,Debian11,standard,alex-smith
3,southcentralus,FALSE,myExistingResourceGroupName,FALSE,,,Ubuntu2204,standard,jan-smith
[empty line for Bash]
```
Note

To be a proper Unix text file and be read by Bash, the CSV file needs a newline character at the end of the last data line. This results in a blank line at the end of the file. Your blank line does not need to say [empty line] as this text is only provided to show you that an empty line exists. PowerShell environments do not have this newline character requirement.
Upload your modified CSV file to your Azure Cloud Shell blog storage account. The easiest way to do this is to use the Manage files drop down on the Azure Cloud Shell main menu. For more information on Cloud Shell storage, see Persist files in Azure Cloud Shell.

Script overview

This article breaks a single large script into four sections so each step can be explained.

Variable setup
Data validation
Loop validation
Azure resource creation

There are also two scripts provided: one for Bash and the second for PowerShell. Both scripts use the same Azure CLI commands. It is the environment, or terminal profile, that is different. For example, Bash uses do...done and if...then...fi. In a PowerShell environment, you use the equivalent foreach and if (something is true)...{do this}. In Azure Cloud Shell you can switch between environments by using the Switch to PowerShell or Switch to Bash button in the Azure Cloud Shell main menu.

If you prefer, go directly to the CSV and script files used by this article in Azure-samples/azure-cli-samples.

Set variables

Get started by creating the variables needed for the script. The following three variables need actual values for your environment:

subscriptionID: This is your Azure subscription ID.
csvFileLocation: This is the location and file name of your CSV input file.
logFileLocation: This is the location and file name the script will use to create a log file. You do not need to create or upload this file.

Variables with a msdocs- prefix can be replaced with the prefix of your choice. All empty ("") variables use values from the CSV input file. These empty variables are placeholders needed by the script.

Bash
PowerShell

# Variable block

# Replace these three variable values with actual values
subscriptionID=00000000-0000-0000-0000-00000000
csvFileLocation="myFilePath\myFileName.csv"
logFileLocation="myFilePath\myLogName.txt"

# Variable values that contain a prefix can be replaced with the prefix of your choice.
#   These prefixes have a random ID appended to them in the script.
# Variable values without a prefix will be overwritten by the contents of your CSV file.
location=""
createRG=""
newRgName="msdocs-rg-"
existingRgName=""

createVnet=""
vnetName="msdocs-vnet-"
subnetName="msdocs-subnet-"
vnetAddressPrefix=""
subnetAddressPrefixes=""

vmName="msdocs-vm-"
vmImage=""
publicIpSku=""
adminUser=""
adminPassword="msdocs-PW-@"

# Set your Azure subscription 
az account set --subscription $subscriptionID

# Variable block

# Replace these three variable values with actual values
$subscriptionID = "00000000-0000-0000-0000-00000000"
$csvFileLocation = "myFilePath\myFileName.csv"
$logFileLocation = "myFilePath\myLogName.txt"

# Variable values that contain a prefix can be replaced with the prefix of your choice.
#   These prefixes have a random ID appended to them in the script.
# Variable values without a prefix will be overwritten by the contents of your CSV file.
$location=""
$createRG=""
$newRgName="msdocs-rg-"
$existingRgName=""

$createVnet=""
$vnetName="msdocs-vnet-"
$subnetName="msdocs-subnet-"
$vnetAddressPrefix=""
$subnetAddressPrefixes=""

$vmName="msdocs-vm-"
$vmImage=""
$publicIpSku=""
$adminUser=""
$adminPassword="msdocs-PW-@"

# Set your azure subscription 
az account set --subscription $subscriptionID

# Import your CSV data
$data = Import-Csv $csvFileLocation -delimiter ","

Validate CSV file values

Before you start to test the create script, make sure your CSV file is formatted correctly and variables will be assigned correct values. This script uses an IF..THEN statement so you can look at one scenario/CSV line at a time.

Bash
PowerShell

# Verify CSV columns are being read correctly

# Take a look at the CSV contents
cat $csvFileLocation

# Validate select CSV row values
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
  # Generate a random ID
  let "randomIdentifier=$RANDOM*$RANDOM"

  # Return the values for the first data row
  # Change the $resourceNo to check different scenarios in your CSV
  if [ "$resourceNo" = "1" ]; then
    echo "resourceNo = $resourceNo"
    echo "location = $location"
    echo "randomIdentifier = $randomIdentifier"
    echo ""
    
    echo "RESOURCE GROUP INFORMATION:"
    echo "createRG = $createRG"
    if [ "$createRG" = "TRUE" ]; then 
      echo "newRGName = $newRgName$randomIdentifier"
    else
      echo "exsitingRgName = $existingRgName"
    fi
    echo ""

    echo "VNET INFORMATION:"
    echo "createVnet = $createVnet"
    if [ "$createVnet" = "TRUE" ]; then 
      echo "vnetName = $vnetName$randomIdentifier"
      echo "subnetName = $subnetName$randomIdentifier"
      echo "vnetAddressPrefix = $vnetAddressPrefix"
      echo "subnetAddressPrefixes = $subnetAddressPrefixes"
    fi
    echo ""

    echo "VM INFORMATION:"
    echo "vmName = $vmName$randomIdentifier"
    echo "vmImage = $vmImage"
    echo "vmSku = $publicIpSku"
    if [ `expr length "$adminUser"` == "1" ]; then
      echo "SSH keys will be generated."
    else
      echo "vmAdminUser = $adminUser"
      echo "vmAdminPassword = $adminPassword$randomIdentifier"        
    fi
  fi  
# skip the header line
done < <(tail -n +2 $csvFileLocation)

# Verify CSV columns are being read correctly

# Take a look at the CSV contents
# The returned table is restricted by the width of your terminal window
$data | Format-Table

# Validate select CSV row values
foreach ($row in $data) {
  $resourceNo = $row.resourceNo
  $location = $row.location
  $createRG = $row.createRG
  $existingRgName = $row.existingRgName
  $createVnet = $row.createVnet
  $vnetAddressPrefix = $row.vnetAddressPrefix
  $subnetAddressPrefixes = $row.subnetAddressPrefixes
  $vmImage = $row.vmImage
  $publicIpSku = $row.publicIpSku
  $adminUser = $row.adminUser
    
  # Generate a random ID
  $randomIdentifier = (New-Guid).ToString().Substring(0,8)

  # Return the values for the first data row
  # Change the $resourceNo to check different scenarios in your CSV
  if ($resourceNo -eq "1") {
    Write-Host "resourceNo = $resourceNo"
    Write-Host "location = $location"
    Write-Host "randomIdentifier = $randomIdentifier"
    Write-Host ""
    
    Write-Host "RESOURCE GROUP INFORMATION:"
    Write-Host "createRG = $createRG"
    if ($createRG -eq "TRUE") {
      Write-Host "newRGName = $newRgName$randomIdentifier"
    }
    else {
        Write-Host "exsitingRgName = $existingRgName"
    }
    Write-Host ""
    
    Write-Host "VNET INFORMATION:"
    Write-Host "createVnet = $createVnet"
    if ($createVnet -eq "TRUE") {
      Write-Host "vnetName = $vnetName$randomIdentifier"
      Write-Host "subnetName = $subnetName$randomIdentifier"
      Write-Host "vnetAddressPrefix = $vnetAddressPrefix"
      Write-Host "subnetAddressPrefixes = $subnetAddressPrefixes"
    }
    Write-Host ""
    
    Write-Host "VM INFORMATION:"
    Write-Host "vmName = $vmName$randomIdentifier"
    Write-Host "vmImage = $vmImage"
    Write-Host "vmSku= $publicIpSku"
    if ($adminUser -ne "") {
      Write-Host "vmAdminUser = $adminUser"
      Write-Host "vmAdminPassword = $adminPassword$randomIdentifier"
      }
    else {
      Write-Host "SSH keys will be generated."
      }
    }
  }

Using the CSV provided in this article, the validation output is as follows: (The 00000001 random ID will be different for each test.)

resourceNo = 1
location = eastus

RESOURCE GROUP INFORMATION:
createRG = TRUE
newRGName = msdocs-rg-00000001

VNET INFORMATION:
createVnet = TRUE
vnetName = msdocs-vnet-00000001
subnetName = msdocs-subnet-00000001
vnetAddressPrefix = 10.0.0.0/16
subnetAddressPrefix = 10.0.0.0/24

VM INFORMATION:
vmName = msdocs-vm-00000001
vmImage = Ubuntu2204
vmSku = standard
SSH keys will be created

Validate script logic

If you are confident in your scripting abilities, you can skip this step. However, because this script is designed to create Azure resources at scale, looping through the script with echo or write-host statements can save you time and unexpected billable Azure resources.

Bash
PowerShell

There are several ways to iterate through a CSV file using Bash. This example uses IFS with a while loop.

# Validate script logic

# Create the log file
echo "SCRIPT LOGIC VALIDATION.">$logFileLocation

# Loop through each row in the CSV file
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
  # Generate a random ID
  let "randomIdentifier=$RANDOM*$RANDOM"
    
  # Log resource number and random ID
  echo "resourceNo = $resourceNo">>$logFileLocation
  echo "randomIdentifier = $randomIdentifier">>$logFileLocation

  # Check if a new resource group should be created
  if [ "$createRG" == "TRUE" ]; then
    echo "Will create RG $newRgName$randomIdentifier.">>$logFileLocation
    existingRgName=$newRgName$randomIdentifier
  fi

  # Check if a new virtual network should be created, then create the VM
  if [ "$createVnet" == "TRUE" ]; then
    echo "Will create VNet $vnetName$randomIdentifier in RG $existingRgName.">>$logFileLocation
    echo "Will create VM $vmName$randomIdentifier in Vnet $vnetName$randomIdentifier in RG $existingRgName.">>$logFileLocation
  else
    echo "Will create VM $vmName$randomIdentifier in RG $existingRgName.">>$logFileLocation
  fi
# Skip the header line.
done < <(tail -n +2 $csvFileLocation)

# Clear the console and display the log file
Clear
cat $logFileLocation

# Validate script logic

# Create the log file
"SCRIPT LOGIC VALIDATION." | Out-File -FilePath $logFileLocation

# Loop through each row in the CSV file
foreach ($row in $data) {
  $resourceNo = $row.resourceNo
  $location = $row.location
  $createRG = $row.createRG
  $existingRgName = $row.existingRgName
  $createVnet = $row.createVnet
  $vnetAddressPrefix = $row.vnetAddressPrefix
  $subnetAddressPrefixes = $row.subnetAddressPrefixes
  $vmImage = $row.vmImage
  $publicIpSku = $row.publicIpSku
  $adminUser = $row.adminUser

  # Generate a random ID
  $randomIdentifier = (New-Guid).ToString().Substring(0,8)
  
  # Log resource number and random ID
  "" | Out-File -FilePath $logFileLocation -Append
  "resourceNo = $resourceNo" | Out-File -FilePath $logFileLocation -Append
  "randomIdentifier = $randomIdentifier" | Out-File -FilePath $logFileLocation -Append
  
  # Check if a new resource group should be created
  if ($createRG -eq "TRUE") {
    "Will create RG $newRgName$randomIdentifier." | Out-File -FilePath $logFileLocation -Append
    $existingRgName = "$newRgName$randomIdentifier"
  }
  
  # Check if a new virtual network should be created, then create the VM
  if ($createVnet -eq "TRUE") {
    "Will create VNet $vnetName$randomIdentifier in RG $existingRgName." | Out-File -FilePath $logFileLocation -Append
    "Will create VM $vmName$randomIdentifier in VNet $vnetName$randomIdentifier in RG $existingRgName." | Out-File -FilePath $logFileLocation -Append
  } else {
    "Will create VM $vmName$randomIdentifier in RG $existingRgName." | Out-File -FilePath $logFileLocation -Append
  }
}

# Display the log file
Get-Content -Path $logFileLocation

Using the CSV provided in this article, the validation output is as follows: (The 00000001, 2, 3 random IDs will be different for each test, but each resource under each resourceNo should share the same random ID.)

resourceNo = 1
createRG = TRUE
createVnet = TRUE
Will create RG msdocs-rg-00000001
Will create VNet msdocs-vnet-00000001 in RG msdocs-rg-00000001
Will create VM msdocs-vm-00000001 within Vnet msdocs-vnet-00000001 in RG msdocs-rg-00000001

resourceNo = 2
createRG = TRUE
createVnet = FALSE
Will create RG msdocs-rg-00000002
Will create VM msdocs-vm-00000002 without Vnet in RG msdocs-rg-00000002

resourceNo = 3
createRG = FALSE
createVnet = FALSE
Will create VM msdocs-vm-00000003 without Vnet in RG <myExistingResourceGroup>

Create Azure resources

You have now created your variable block, validated your CSV values, and completed a test run with echo or write-host. Execute the fourth and final portion of the script to create Azure resources as defined in your CSV input file.

Bash
PowerShell

# Create Azure resources

# Create the log file
echo "CREATE AZURE RESOURCES.">$logFileLocation

# Loop through each CSV row
while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
  # Generate a random ID
  let "randomIdentifier=$RANDOM*$RANDOM"

  # Log resource number, random ID and display start time
  echo "resourceNo = $resourceNo">>$logFileLocation
  echo "randomIdentifier = $randomIdentifier">>$logFileLocation
  echo "Starting creation of resourceNo $resourceNo at $(date +"%Y-%m-%d %T")."

  # Check if a new resource group should be created
  if [ "$createRG" == "TRUE" ]; then
    echo "Creating RG $newRgName$randomIdentifier at $(date +"%Y-%m-%d %T").">>$logFileLocation
    az group create --location $location --name $newRgName$randomIdentifier >>$logFileLocation
    existingRgName=$newRgName$randomIdentifier
    echo "  RG $newRgName$randomIdentifier creation complete"
  fi

  # Check if a new virtual network should be created, then create the VM
  if [ "$createVnet" == "TRUE" ]; then
    echo "Creating VNet $vnetName$randomIdentifier in RG $existingRgName at $(date +"%Y-%m-%d %T").">>$logFileLocation
    az network vnet create \
        --name $vnetName$randomIdentifier \
        --resource-group $existingRgName \
        --address-prefix $vnetAddressPrefix \
        --subnet-name $subnetName$randomIdentifier \
        --subnet-prefixes $subnetAddressPrefixes >>$logFileLocation
    echo "  VNet $vnetName$randomIdentifier creation complete"
    
    echo "Creating VM $vmName$randomIdentifier in Vnet $vnetName$randomIdentifier in RG $existingRgName at $(date +"%Y-%m-%d %T").">>$logFileLocation
    az vm create \
        --resource-group $existingRgName \
        --name $vmName$randomIdentifier \
        --image $vmImage \
        --vnet-name $vnetName$randomIdentifier \
        --subnet $subnetName$randomIdentifier \
        --public-ip-sku $publicIpSku \
        --generate-ssh-keys >>$logFileLocation
    echo "  VM $vmName$randomIdentifier creation complete"
  else
    echo "Creating VM $vmName$randomIdentifier in RG $existingRgName at $(date +"%Y-%m-%d %T").">>$logFileLocation
    az vm create \
        --resource-group $existingRgName \
        --name $vmName$randomIdentifier \
        --image $vmImage \
        --public-ip-sku $publicIpSku \
        --admin-username $adminUser\
        --admin-password $adminPassword$randomIdentifier >>$logFileLocation
    echo "  VM $vmName$randomIdentifier creation complete"    
  fi
# skip the header line
done < <(tail -n +2 $csvFileLocation)

# Clear the console (optional) and display the log file
# clear
cat $logFileLocation

# Create Azure resources

# Create the log file
"CREATE AZURE RESOURCES." | Out-File -FilePath $logFileLocation

# Loop through each CSV row
foreach ($row in $data) {
  $resourceNo = $row.resourceNo
  $location = $row.location
  $createRG = $row.createRG
  $existingRgName = $row.existingRgName
  $createVnet = $row.createVnet
  $vnetAddressPrefix = $row.vnetAddressPrefix
  $subnetAddressPrefixes = $row.subnetAddressPrefixes
  $vmImage = $row.vmImage
  $publicIpSku = $row.publicIpSku
  $adminUser = $row.adminUser
    
  # Generate a random ID
  $randomIdentifier = (New-Guid).ToString().Substring(0,8)
  
  # Log resource number, random ID and display start time
  "" | Out-File -FilePath $logFileLocation -Append
  "resourceNo = $resourceNo" | Out-File -FilePath $logFileLocation -Append
  "randomIdentifier = $randomIdentifier" | Out-File -FilePath $logFileLocation -Append
  Write-Host "Starting creation of resourceNo $resourceNo at $(Get-Date -format 'u')."
  
  # Check if a new resource group should be created
  if ($createRG -eq "TRUE") {
    "Creating RG $newRgName$randomIdentifier at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
    az group create --location $location --name $newRgName$randomIdentifier | Out-File -FilePath $logFileLocation -Append
    $existingRgName = "$newRgName$randomIdentifier"
    Write-Host "  RG $newRgName$randomIdentifier creation complete."
    }
  
  # Check if a new virtual network should be created, then create the VM
  if ($createVnet -eq "TRUE") {
    "Creating VNet $vnetName$randomIdentifier in RG $existingRgName at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
    az network vnet create `
        --name $vnetName$randomIdentifier `
        --resource-group $existingRgName `
        --address-prefix $vnetAddressPrefix `
        --subnet-name $subnetName$randomIdentifier `
        --subnet-prefixes $subnetAddressPrefixes | Out-File -FilePath $logFileLocation -Append
    Write-Host "  VNet $vnetName$randomIdentifier creation complete."
    
    "Creating VM $vmName$randomIdentifier in Vnet $vnetName$randomIdentifier in RG $existingRgName at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
    az vm create `
        --resource-group $existingRgName `
        --name $vmName$randomIdentifier `
        --image $vmImage `
        --vnet-name $vnetName$randomIdentifier `
        --subnet $subnetName$randomIdentifier `
        --public-ip-sku $publicIpSku `
        --generate-ssh-keys | Out-File -FilePath $logFileLocation -Append
    Write-Host "  VM $vmName$randomIdentifier creation complete." 
  } else {
    "Creating VM $vmName$randomIdentifier in RG $existingRgName at $(Get-Date -format 'u')." | Out-File -FilePath $logFileLocation -Append
    az vm create `
        --resource-group $existingRgName `
        --name $vmName$randomIdentifier `
        --image $vmImage `
        --public-ip-sku $publicIpSku `
        --admin-username $adminUser `
        --admin-password $adminPassword$randomIdentifier | Out-File -FilePath $logFileLocation -Append
    Write-Host "  VM $vmName$randomIdentifier creation complete."
  }
}

# Clear the console (optional) and display the log file
# Clear-Host
Get-Content -Path $logFileLocation

Console output before log file read:

Starting creation of resourceNo 1 at YYYY-MM-DD HH:MM:SS.
  RG msdocs-rg-00000001 creation complete
  VNet msdocs-vnet-00000001 creation complete
  VM msdocs-vm-00000001 creation complete
Starting creation of resourceNo 2 at YYYY-MM-DD HH:MM:SS.
  RG msdocs-rg-00000002 creation complete
  VM msdocs-vm-00000002 creation complete
Starting creation of resourceNo 3 at YYYY-MM-DD HH:MM:SS.
  VM msdocs-vm-00000003 creation complete

Troubleshooting

Bash script ignores IF statement

Bash is case sensitive. The word true does not equal TRUE. Also greater than is -gt, not >, and equals is ==, not =. Make sure you do not have a typographical error, or leading/trailing spaces in your CSV column values.

Variable values are not changing with each loop

This is often caused by extra spaces in the CSV file. A line in a CSV file will look something like this: column1,column2,column3 or column1,,column3, but by habit it is easy to create a test file that contains a space after each comma like column1, column2, column3. When you have a leading or trailing space in your CSV, the column value is actually <space>columnValue. The script logic if [ "$columnName" = "columnValue" ] returns "false". Remove all leading and trailing spaces in your CSV rows to fix the issue.

Invalid CIDR notation

You receive an InvalidCIDRNotation error when you pass an incorrect address prefix to az network vnet create. This can be challenging when visually, the address prefix looks correct when returned in an echo statement. To troubleshoot the actual value being read from the CSV, try this script:

while IFS=, read -r resourceNo location createRG existingRgName createVnet vnetAddressPrefix subnetAddressPrefixes vmImage publicIpSku adminUser
do
    echo "resourceNo = $resourceNo"

    if [ "$createVnet" == "TRUE" ]; then
      startTest="abc"
      endTest="xyz"
      echo $startTest$vnetAddressPrefix$endTest
    fi
done < <(tail -n +2 $setupFileLocation)

If your results look like xzy10.0.0.0 and not the expected abc10.0.0.0/24xyz, there might be a hidden character or extra comma lurking in your CSV file. Add a test column with the same prefix value, rearrange your CSV columns, and copy/paste your CSV contents in/out of a simple Notepad editor. In writing this article, the rearrangement of the CSV columns finally fixed the error.

Arguments are expected or required

You receive this error when you have not supplied a required parameter or there is a typographical error that causes the Azure CLI to incorrectly parse the reference command. When working with a script, you also receive this error when one of more of the following is true:

There is a missing or incorrect line continuation character.
There are trailing spaces on the right side of a line continuation character.
Your variable name contains a special character, such as a dash (-).

InvalidTemplateDeployment

When you try to create an Azure resource in a location that does not offer that resource you receive an error similar to the following: "Following SKUs have failed for Capacity Restrictions: Standard_DS1_v2' is currently not available in location 'westus'."

Here's the full error example:

{"error":{"code":"InvalidTemplateDeployment","message":"The template deployment 'vm_deploy_<32 character ID>'
is not valid according to the validation procedure. The tracking id is '<36 character ID>'.
See inner errors for details.","details":[{"code":"SkuNotAvailable","message":"The requested VM size for resource
'Following SKUs have failed for Capacity Restrictions: Standard_DS1_v2' is currently not available
in location '<your specified location>'. Please try another size or deploy to a different location
or different zone. See https://aka.ms/azureskunotavailable for details."}]}}

To correct the error, either change the location or select a different parameter value that is offered for your desired location.

Share via