Easily Multi thread Powershell commands and scripts

  • Makale

Update: I had recreated the same functionality in C# using Tasks. The Powershell binary module has some added features when compared with the below PS Version.

Download the powershell binary module from Github repo.

If you require the PS1 version of the script, download it here.
invokeallV2.5.renametoPS1
Rename the file to .ps1 to use it on the PS window

I want to share this script, I wrote for my customer few months back. This one is special as it is my first script without any Exchange commands in it :-) .

There are many everyday tasks in support and administration which takes long time to complete. Some of the common tasks are

  •                Collecting asset information, configuration from all the workstations
  •                Collect mailbox statistics or other details for thousands of objects...etc.

I had seen some commands/scripts that was executed for large number of objects and took more than 24 hours to complete. Scripting is all about reducing time spent performing mundane tasks, but seems it is not enough. The commands simply take so long just because of too many objects it has to process sequentially. Then comes the idea of Multi-threading in powershell, using powershell background runspaces seem fit for the job than using Powershell jobs. Background Runspaces are more faster and light weight to process more number of tasks in parallel.

I started integrating runspaces to achieve multi-threading on my scripts, but for every script, the requirement is different and coding it consumes time. To overcome this challenge, I decided to create a generic function that can be used on all the commands and scripts with very less modification to the actual command. Not all Administrators who uses powershell for the daily administrative tasks and reporting are well versed on scripting. With all that kept in mind, Invoke-all function was created, I have made the function very generic, easy-to-use and lightweight as possible.

The script can be downloaded from https://aka.ms/invokeall

Usage example:

#Actual command:
[powershell]
Get-Mailbox -database 'db1' | foreach{ Get-MailboxFolderStatistics -Identity $_.name -FolderScope "inbox" -IncludeOldestAndNewestItems } | Where-Object{$_.ItemsInFolder -gt 0} | Select Identity, Itemsinfolder, Foldersize, NewestItemReceivedDate
[/powershell]
Same command with Invoke-all
[powershell]
Get-Mailbox -database 'db1' | Invoke-All{ Get-MailboxFolderStatistics -Identity $_.name -Folder Scope "inbox" -IncludeOldestAndNewestItems }  | Where-Object{$_.ItemsInFolder -gt 0} | Select Identity, Itemsinfolder, Foldersize, NewestItemReceivedDate
[/powershell]
As you see it is that easy to use. Imagine, 100s of DBs and using invoke-all function is 10x faster than the conventional way.

Note: The scriptblock is the first parameter to the Invoke-all function and it should only contain the first command and its parameters that needs to be executed. Any Filtering that is done using another command (Where-Object) must be outside of the script block as mentioned in the above example.

Features:

  1. Easy to use.
  2. When used with Cmdlets or Remote PowerShell, it automatically detects and loads the (only) required modules to the sessions.
  3. Checks if it can complete the first job without errors before scheduling all the jobs to run in parallel.
  4. By default, the script processes the jobs in batches. With batching, the resource consumption is very less and its proportional to the batch size which can be adjusted as required.
  5. Ability to pause the Job scheduling in-between batches. This is very useful if too many requests are overloading the destination.
  6. Creates a runtime log by default.

The Invoke-all is a function exported and is a wrapper like function that takes input from Pipeline or using explicit parameter and process the command concurrently using runspaces. I have mainly focused on Exchange On-Perm (and some Online Remote powershell (RPS) commands) while developing the script. Though I have only tested it with Exchange On-Perm Remote sessions and Snap-ins, it can be used on any powershell cmdlet or function.

I haven’t tested it with any Set-* commands, but it is possible to use this script to modify objects in parallel. Carefully chose the command so that there is no contention created when using multiple threads.

Exchange remote PowerShell uses Implicit remoting, the Proxyfunctions used by the Remote session doesn’t accept value from pipeline. Therefore, the Script requires you to declare all the Parameters that you want to use in the command when using such commands. See examples on how to do it.

It can handle cmdlets and functions from any modules. I have updated script to handle external scripts as well. I will see to include applications handling in the next version, but it can be achieved in this version with a simple script as shown in the examples that can execute your EXE or application.

Please refer to the comments on the script to learn more about the technique used on the script or please leave a comment here. Hope you find this one useful to save some time on your long running powershell commands.

Comments

  • Anonymous
    October 16, 2017
    Thanks alotim a fan from ver 1.0
    • Anonymous
      October 16, 2017
      Thank you Turbo, glad it is of use :-)
  • Anonymous
    October 12, 2018
    good idea .
  • Anonymous
    January 04, 2019
    Can you post your code again? it is missing from technet gallery....
    • Anonymous
      January 07, 2019
      please find the link to the script on top of this post, thanks
  • Anonymous
    February 20, 2019
    Hi,I love the idea, but I can't get it to work. I'm trying to pass to it a block of lines from a set of files, on which my own script needs to do some regex matching and writes results to a MongoDB. I'm getting an index out of bounds error. Is there a limit to the amount of objects that can be passed to the invokeAll command?Tx.
    • Anonymous
      February 20, 2019
      It uses an array for input object, I have tested with more than 300k input objects and it does work fine..can you provide more details and share me the log file the script creates..tx
      • Anonymous
        February 20, 2019
        The comment has been removed
        • Anonymous
          February 20, 2019
          I'm sorry but the layout above is removed by the blog page, I'm afraid... I also tried feeding this script a directory with smaller files, but it was the same error...Powershell really needs this kind of functionality and not just in workflows imho. I've looked into converting my script(s) into workflows, but there's quite some differences and therefore quite some work. Great job in coding and many thanks for sharing!
        • Anonymous
          February 21, 2019
          The comment has been removed
          • Anonymous
            February 22, 2019
            The comment has been removed
            • Anonymous
              February 22, 2019
              The comment has been removed
          • Anonymous
            February 22, 2019
            The comment has been removed
            • Anonymous
              February 23, 2019
              As you mentioned, going thru each file is the best way, pl see if you can get it to work (good luck!!).. Am working now to rewrite the script using c# which will have the functionality you mentioned, process x number of jobs and Queue the remaining as they complete in an efficient way. I will post here when I complete it
          • Anonymous
            February 27, 2019
            Thanks again for the assistance and the (sharing of your) code. I ended up rewriting everything, splitting files in arrays of 500 each and got it to work with Invoke-All, but was still bothered by the overhead of creating all the jobs beforehand. Finally I ended up creating my own runspacefactory with runspaces, tweaked to run this specific code, ending up in a performance increase of 800%! Should your generic C# script be ready, I'm interested in having a look. Th@nks!
  • Anonymous
    March 11, 2019
    Hello Santhosh Sethumadhavan,Great script! It really enables me to do great things.However, please change the PowerShell versioncheck in the script in the next update:Current: $host.Version.MajorImprovement: $PSVersionTable.PSVersion.MajorThe issue with $host is that it will return version "1" if you're connected to a remote host with PowerShell Remoting. The $PSVersionTable will display the actual PowerShell version of the remote host.
    • Anonymous
      March 12, 2019
      Thank you for the suggestion Patrick. I will add it on the next version.