Implement multi-threading with .NET runspaces in Powershell

There are multiple options on how to implement multi-threading in Powershell. They are all well known to engineers:

• Powershell Jobs
• Powershell Workflows

But there is another one, which is not quite popular (because of it's complexity), but very powerful: .NET runspaces. While it's quite difficult to implement, it don't have main disadvantage of native Powershell ways - we will not spawn tons of powershell.exe processes. All work will be done within a single process and that will highly increase overall script performance.

I will not tell you about theory (mainly because I don't clearly understand details), but will give you some insights and script templates to start moving.

What I know - is that runspace is the single space to invoke some code, while runspacepool - is the pool for the multiple runspaces and .NET knows how to aumatically manage them.

So, let's start.

initialSessionState will hold typeDatas and functions that will be passed to every runspace.

$initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault();

Now we'll define some function.

function getSomeThing {
	param (
		[Parameter(Mandatory=$true)][string]$url
	);
	try {
		$doc = Invoke-WebRequest $url -Method Get -DisableKeepAlive -ErrorAction SilentlyContinue;		
	} catch [System.Net.WebException] {
		Write-Host "$url : Invoke-WebRequest Error";
		return $false;
	}
	
	return $doc;
}
$getSomeThing_def = Get-Content Function:\getSomeThing;
$getSomeThing_SessionStateFunction = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList 'getSomeThing', $getSomeThing_def;
$initialSessionState.Commands.Add($getSomeThing_SessionStateFunction);

In last three strings we're adding our function to the initialSessionState. Ok, maybe you like object-oriented approach as I do, then here's how you'll define your TypeData and add it to session state:

$init = @{
	MemberName = 'Init';
	MemberType = 'ScriptMethod';
	Value = {
		Add-Member -InputObject $this -MemberType NoteProperty -Name Url -Value $null;
		Add-Member -InputObject $this -MemberType NoteProperty -Name Title -Value $null;
	};
	Force = $true;
}

$populate = @{
	MemberName = 'Populate';
	MemberType = 'ScriptMethod';
	Value = {
		param (
			[Parameter(Mandatory=$true)][string]$url
		);
		$this.url = $url;
		$this.title = getSomeThing($url);
	};
	Force = $true;
}

Update-TypeData -TypeName 'Custom.Object' @Init;
Update-TypeData -TypeName 'Custom.Object' @Populate;
$customObject_typeEntry = New-Object System.Management.Automation.Runspaces.SessionStateTypeEntry -ArgumentList $(Get-TypeData Custom.Object), $false;
$initialSessionState.Types.Add($customObject_typeEntry);

 Next we'll define our main, entry point to runspace.

$ScriptBlock = {
	Param (
		[PSCustomObject]$url
	)
   
	$page = [PsCustomObject]@{PsTypeName ='Custom.Object'};
	$page.Init();
    $page.Populate($url);
   
	$Result = New-Object PSObject -Property @{
		title = $null
		url = $url
	};
		
	
    return $Result;
}

And - finally - we're going to spin things up.

$Throttle = 15; #threads

$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1, $Throttle, $initialSessionState, $Host);
$RunspacePool.Open();
$Jobs = @();

$i = 0;
foreach($url in $urls) { #$urls - some array of URLs
	$i++;
	
	$Job = [powershell]::Create().AddScript($ScriptBlock).AddArgument($url);
	$Job.RunspacePool = $RunspacePool;
	$Jobs += New-Object PSObject -Property @{
	   RunNum = $i;
	   Pipe = $Job;
	   Result = $Job.BeginInvoke();
	}
}

$results = @();
foreach ($Job in $Jobs) {
	$Results += $Job.Pipe.EndInvoke($Job.Result);
}

Results are collected in the very end.

That's it.