PSParallelPipeline

_{Parallel processing of pipeline input objects!}

PSParallelPipeline is a PowerShell Module that includes Invoke-Parallel, a cmdlet that allows for parallel processing of input objects, sharing similar capabilities as ForEach-Object -Parallel introduced in PowerShell v7.0.

This project was inspired by RamblingCookieMonster's Invoke-Parallel and is developed with Windows PowerShell 5.1 users in mind where the closest there is to parallel pipeline processing is Start-ThreadJob.

What does this Module offer?

Except for -AsJob, this module offers the same capabilities as ForEach-Object -Parallel in addition to supporting Common Parameters, a missing feature in the built-in cmdlet.

Pipeline streaming capabilities

Measure-Command {
    $null | Invoke-Parallel { 0..10 | ForEach-Object { Start-Sleep 1; $_ } } |
        Select-Object -First 1
} | Select-Object TotalSeconds

# TotalSeconds
# ------------
#        1.06

Support for CommonParameters

Something missing on ForEach-Object -Parallel as of v7.5.0.3.

PS \> 0..5 | ForEach-Object -Parallel { Write-Error $_ } -ErrorAction Stop
# ForEach-Object: The following common parameters are not currently supported in the Parallel parameter set:
# ErrorAction, WarningAction, InformationAction, PipelineVariable

A few examples, they should all work properly, please submit an issue if not 😅.

PS \> 0..5 | Invoke-Parallel { Write-Error $_ } -ErrorAction Stop
# Invoke-Parallel: 0

PS \>  0..5 | Invoke-Parallel { Write-Warning $_ } -WarningAction Stop
# WARNING: 1
# Invoke-Parallel: The running command stopped because the preference variable "WarningPreference" or common parameter is set to Stop: 1

PS \> 0..5 | Invoke-Parallel { $_ } -PipelineVariable pipe | ForEach-Object { "[$pipe]" }
# [0]
# [1]
# [5]
# [2]
# [3]
# [4]

Improved -TimeOutSeconds error message

In ForEach-Object -Parallel we get an error message per stopped parallel invocation instead of a single one.

PS \>  0..10 | ForEach-Object -Parallel { $_; Start-Sleep 5 } -TimeoutSeconds 2
# 0
# 1
# 2
# 3
# 4
# InvalidOperation: The pipeline has been stopped.
# InvalidOperation: The pipeline has been stopped.
# InvalidOperation: The pipeline has been stopped.
# InvalidOperation: The pipeline has been stopped.
# InvalidOperation: The pipeline has been stopped.

With Invoke-Parallel you get a single, friendlier, error message.

PS \> 0..10 | Invoke-Parallel { $_; Start-Sleep 5 } -TimeoutSeconds 2
# 0
# 1
# 2
# 3
# 4
# Invoke-Parallel: Timeout has been reached.

$using: Support

Same as ForEach-Object -Parallel you can use the $using: scope modifier to pass-in variables to the parallel invocations.

$message = 'world!'
'hello ' | Invoke-Parallel { $_ + $using:message }
# hello world!

-Functions and -Variables Parameters

Both parameters are a quality of life addition, specially -Functions, which adds the locally defined functions to the runspaces Initial Session State, a missing feature on ForEach-Object -Parallel. This is a much better alternative to passing-in the function definition to the parallel scope.

-Variables Parameter

'hello ' | Invoke-Parallel { $_ + $msg } -Variables @{ msg = 'world!' }
# hello world!

-Functions Parameter

function Get-Message {param($MyParam) $MyParam + 'world!' }
'hello ' | Invoke-Parallel { Get-Message $_ } -Functions Get-Message
# hello world!

Documentation

Check out the docs for information about how to use this Module.

Installation

Gallery

The module is available through the PowerShell Gallery:

Install-Module PSParallelPipeline -Scope CurrentUser

Source

git clone 'https://github.com/santisq/PSParallelPipeline.git'
Set-Location ./PSParallelPipeline
./build.ps1

Requirements

This module has no requirements and is fully compatible with Windows PowerShell 5.1 and PowerShell Core 7+.

Contributing

Contributions are more than welcome, if you wish to contribute, fork this repository and submit a pull request with the changes.