PowerShell workflows in Azure Automation with parallel processing of Azure subscriptions

Question

I am trying to utilise 'foreach -Parallel' to perform processing across multiple subscriptions however I've noticed that 'parallel' uses the same 'PSSourceJobInstanceId' as the workflow therefore only supports one subscription context at a time.

Is there a work-around for this or do Azure Automation workflows not support 'Parallel' across subscription contexts?

Code example;

workflow WF_Test
{
    $cred = Get-AutomationPSCredential -Name '###'
    $null = Add-AzAccount -Credential $cred
    $subscriptionList = @('ec5f9429-###','ccd47b39-###')
    "* With -Parallel"
    foreach -Parallel ($subscription in $subscriptionList) {
        Set-AzContext -Subscription $subscription
        "Total VMs $((Get-AzVM).Count)"
    }
    "* Without -Parallel"
    Set-AzContext -Subscription $subscriptionList[0]
    "Total VMs $((Get-AzVM).Count)"
    Set-AzContext -Subscription $subscriptionList[1]
    "Total VMs $((Get-AzVM).Count)"
}

Output;

With -Parallel

PSComputerName        : localhost
PSSourceJobInstanceId : 2b5feb08-dadb-4b21-a689-aa94f169095b
Name                  : NonProd (ccd47b39-###) - ### - ###
Account               : ###
Environment           : AzureCloud
Subscription          : ccd47b39-###
Tenant                : ###
TokenCache            : Microsoft.Azure.Commands.Common.Authentication.Core.ProtectedFileTokenCache
VersionProfile        :
ExtendedProperties    : {}
PSComputerName        : localhost

Total VMs - 98

PSSourceJobInstanceId : 2b5feb08-dadb-4b21-a689-aa94f169095b
Name                  : Prod (ec5f9429-###) - ### - ###
Account               : ###
Environment           : AzureCloud
Subscription          : ec5f9429-###
Tenant                : ###
TokenCache            : Microsoft.Azure.Commands.Common.Authentication.Core.ProtectedFileTokenCache
VersionProfile        :
ExtendedProperties    : {}

Total VMs - 98

Without -Parallel

PSComputerName        : localhost
PSSourceJobInstanceId : 2b5feb08-dadb-4b21-a689-aa94f169095b
Name                  : Prod (ec5f9429-###) - ### - ###
Account               : ###
Environment: AzureCloud
Subscription          : ec5f9429-###
Tenant                : ###
TokenCache            : Microsoft.Azure.Commands.Common.Authentication.Core.ProtectedFileTokenCache
VersionProfile        :
ExtendedProperties    : {}

Total VMs - 98

PSComputerName        : localhost
PSSourceJobInstanceId : 2b5feb08-dadb-4b21-a689-aa94f169095b
Name                  : NonProd (ccd47b39-###) - ### - ###
Account               : ###
Environment           : AzureCloud
Subscription          : ccd47b39-###
Tenant                : ###
TokenCache            : Microsoft.Azure.Commands.Common.Authentication.Core.ProtectedFileTokenCache
VersionProfile        :
ExtendedProperties    : {}

Total VMs - 31

Also tried using a 'runspace' withing the 'foreach -Parallel' however this made no difference;

        InlineScript {
            $Runspace = [runspacefactory]::CreateRunspace()
            $PowerShell = [powershell]::Create()
            $PowerShell.runspace = $Runspace
            $Runspace.Open()
            [void]$PowerShell.AddScript({
                Set-AzContext -Subscription $Using:subscription
                "Total VMs $((Get-AzVM).Count)"
            })
            $PowerShell.Invoke()
            $PowerShell.Dispose()
        }

Thanks in advance.

Paul

Accepted Answer

Thanks for your insights, @Stanislav Zhelyazkov .

@Asperia ,

Great analysis! and thanks for the detailed steps provided! This looks like a feature gap so if interested, I would recommend you to raise this as a feature request in this UserVoice or feedback forum and also recommend you to upvote it and other features that are of interest. In general, Azure product / feature team would check feasibility of a feature request, triage it, prioritize against existing feature backlog, add in roadmap as appropriate and would announce and/or update the related Azure document once a feature request is addressed.

On the other hand, I don’t have a handy setup with multiple subscriptions to quickly test this out, but see if the below explained approach works as a workaround.

As described here, try to leverage child runbook so create 2 runbooks i.e., one parent and one child where parent runbook has foreach -parallel block and calls the child runbook using Start-AzAutomationRunbook cmdlet as shown below. And then child runbook does get the count of number of VM’s.

Parent runbook sample:

workflow PSWF1  
{  
  
    #...  
    #Add your automation connection code here  
    #...  
  
    $subscriptionList = @('xxxxxxxxxxxxxxxxxxxxxxxxxxxxx','yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy')  
    $ChildRunbookInputParams = @{"SubscriptionId"="$($sub.SubscriptionId)"}  
    " With -Parallel"  
    foreach -Parallel ($subscription in $subscriptionList) {          
        $job = Start-AzAutomationRunbook -Name "PSWF1ChildRunbook" -Parameters @{"SubscriptionId"="$subscription"} -ResourceGroupName "" -AutomationAccountName ""  
    }  
}

Child runbook sample:

workflow PSWF1ChildRunbook  
{  
    Param  
    (  
    [Parameter (Mandatory= $true)]  
    [String] $SubscriptionId  
    )  
      
    #...  
    #Add your automation connection code here  
    #...  
      
    $SubscriptionContext = Set-AzContext -Subscription $SubscriptionId  
    "Total VMs $((Get-AzVM -AzContext $SubscriptionContext).Count)"  
}

On publishing both the runbooks and then running the parent runbook, it should trigger two separate jobs for your two subscriptions under the child runbook.

Answer

Hi,
This is not possible. Context is being shared even among two separate jobs in the same automation account.

P.s.do not forget to mark this reply as answer.

Answer

Hi Stan, thanks for your answer. However, from my testing workflows from the same Automation Account can run concurrently under different subscription contexts as shown below. Therefore it seems the answer is that an AA job will only support one subscription context at a time, including within parallel processing i.e. Mulitple concurrent Azure subscriptions contexts are not supported for workflow parallel processing :(

The following example shows two jobs running in the same AA at the same time with different subscription contexts;

workflow WF_Subscription1
{
$cred = Get-AutomationPSCredential -Name 'CoGc'
$null = Add-AzAccount -Credential $cred
$subscriptionList = @('ec5f9429-###','ccd47b39-###')
Set-AzContext -Subscription $subscriptionList[0]
"$((Get-Date).ToString()) Subscription 'ec5f9429': Total VMs $((Get-AzVM).Count)"
sleep 60
"$((Get-Date).ToString()) Subscription 'ec5f9429': Total VMs $((Get-AzVM).Count)"
}
Output:
PSComputerName : localhost
PSSourceJobInstanceId : c87fee9f-5107-48b8-aac0-7ab75f29db63
Name : Prod (ec5f9429-###) - ### - ###
Account : ###
Environment : AzureCloud
Subscription : ec5f9429-###
Tenant : ###
9/21/2020 9:36:30 AM Subscription 'ec5f9429': Total VMs 98
9/21/2020 9:37:34 AM Subscription 'ec5f9429': Total VMs 98

workflow WF_Subscription2
{
$null = Disable-AzContextAutosave -Scope Process
$cred = Get-AutomationPSCredential -Name 'CoGc'
$null = Add-AzAccount -Credential $cred
$subscriptionList = @('ec5f9429-###’,'ccd47b39-###’)
Set-AzContext -Subscription $subscriptionList[1]
"$((Get-Date).ToString()) Subscription 'ccd47b39': Total VMs $((Get-AzVM).Count)"
sleep 60
"$((Get-Date).ToString()) Subscription 'ccd47b39': Total VMs $((Get-AzVM).Count)"
}
Output:
PSComputerName : localhost
PSSourceJobInstanceId : e463d58c-bbaa-4c5b-b942-9013b0b11533
Name : NonProd (ccd47b39-###) - ### - ###
Account : ###
Environment : AzureCloud
Subscription : ccd47b39-###
Tenant : ###
9/21/2020 9:35:59 AM Subscription 'ccd47b39': Total VMs 31
9/21/2020 9:37:01 AM Subscription 'ccd47b39': Total VMs 31

Cheers,
Paul

Share via

PowerShell workflows in Azure Automation with parallel processing of Azure subscriptions

2 additional answers