Contacts import using runspacepools

Question

I'm trying to import contacts using RunspacePools, but I'm having trouble getting it to work. If I take it out of the runspace logic, it works fine, just takes a long time. I'd really like to use runspacepools to speed up the import process and make it run multithreaded so it imports faster. On avg each import takes about 5-6 mins per user, and I have about 500 users, so it can take up to 3000 mins to run.

Here is what I currently have:

#---------------------------------------------
. $rootPath\UpdateContacts\UpdateContacts.ps1

# Set up runspace pool
$RunspacePool = [runspacefactory]::CreateRunspacePool(1,10)
$RunspacePool.Open()

# Assign new jobs/runspaces to a variable
$Runspaces = foreach ($User in $Users)
{
            
    # Create new PowerShell instance to hold the code to execute, add arguments
    $PSInstance = [powershell]::Create().AddScript({
        $Users | ForEach{ UpdateContacts($_) }
    }).AddParameter('$_')

    # Assing PowerShell instance to RunspacePool
    $PSInstance.RunspacePool = $RunspacePool

    # Start executing asynchronously, keep instance + IAsyncResult objects
    New-Object psobject -Property @{
        Instance = $PSInstance
        IAResult = $PSInstance.BeginInvoke()
        Argument = $User
    }
}

# Wait for the the runspace jobs to complete
while($Runspaces |Where-Object{-not $_.IAResult.IsCompleted})
{
    Start-Sleep -Milliseconds 500
}

# Collect the results
$Results = $Runspaces |ForEach-Object {
    $Output = $_.Instance.EndInvoke($_.IAResult)
    
    New-Object psobject -Property @{
        User = $User
    }
}

And my "UpdateContacts.ps1" file looks like this:

Function UpdateContacts($User)
{
        Write-host "Importing Contacts. This could take several minutes."

        #FirstName, MiddleName, LastName, DisplayName, SamAccountName, Email, Mobile, TelephoneNumber, Title, Dept, Company, Photo, ExtensionAttribute2
        $ContactsBody = @"
        { 
            "givenName"      : "$($User.FirstName)",
            "middleName"     : "$($User.MiddleName)",
            "surname"        : "$($User.LastName)",
            "displayName"    : "$($User.DisplayName)",
            "jobTitle"       : "$($User.Title)",
            "companyName"    : "$($User.Company)",
            "department"     : "$($User.Dept)",
            "mobilePhone"    : "$($User.Mobile)",
            "homePhones"     : ["$($User.TelephoneNumber)"],
            "emailAddresses": 
            [
                {
                    "address": "$($User.Email)",
                    "name": "$($User.DisplayName)"
                }
            ]
        }
"@
        Try
        {
            Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId/contacts" -Headers $headers -Body $ContactsBody -Method Post -ContentType 'application/json' | Out-Null

            #After each user clear the info
            $User = $NULL
        }
        Catch
        {
            if($error)
            {
                $User
                $error
                pause
            }

            $_.Exception.Message
            Write-Host "--------------------------------------------------------------------------------------"
            $_.Exception.ItemName
        }
}

Any help is appreciated.

EDIT: Here is the full script (with the exception of the ContactUploader.ps1 script. That function is in a separate script but the whole code (Function) is posted above).

CLS

##################### Import Thread/Job Module to perform multithreading #####################

if(!(Get-Module -ListAvailable -Name ThreadJob))
{
    $NULL = Install-Module -Name ThreadJob -Scope CurrentUser -Force -Confirm:$False
}

##################### ------------------------------------------------------------- #####################

#Root Path
$rootPath = $(Split-path $MyInvocation.MyCommand.path -Parent)

#Prevent connection from closing on us when we use "Invoke-RestMethod"
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls -bor [Net.SecurityProtocolType]::Tls11 -bor [Net.SecurityProtocolType]::Tls12

Add-Content "$rootPath\progress.txt" ""
Add-Content "$rootPath\progress.txt" "********** Starting Script $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

##################### Connect to Microsoft Graph API and configure all of our Variables #####################

Add-Content "$rootPath\progress.txt" "********** Connecting to Graph API $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

$ApplicationID = "ApplicationID"
$TenatDomainName = "domain.com"
$AccessSecret = "ItsASecret"

$global:Body = @{    
Grant_Type    = "client_credentials"
Scope         = "https://graph.microsoft.com/.default"
client_Id     = $ApplicationID
Client_Secret = $AccessSecret
} 

$ConnectGraph = Invoke-RestMethod -Uri "https://login.microsoftonline.com/$TenatDomainName/oauth2/v2.0/token" -Method POST -Body $Body

Add-Content "$rootPath\progress.txt" "********** Finished Connecting to Graph API $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

$global:token = $ConnectGraph.access_token

$global:UPN = "user@domain.com"
$global:AccessToken = $token
$global:User = $NULL
$global:Contact = $NULL
$global:NeedsToBeAdded = $NULL
$global:NeedsToBeDeleted = $NULL
$global:folderId = $NULL
$global:NewContactFolder = $NULL

$global:FolderName = "Test Contacts"

$global:headers = @{
            "Authorization"    = "Bearer $AccessToken"
            "Accept"           = "application/json;odata.metadata=none"
            "Content-Type"     = "application/json; charset=utf-8"
            "ConsistencyLevel" = "eventual"
}

#Create Contact Folder if it doesn't exist
$global:ContactsFolderBody = @"
    { 
        "parentFolderId": "$ParentFolderID",
        "displayName": "Test Contacts"
    }
"@

Add-Content "$rootPath\progress.txt" "********** Grabbing Contact Folder Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

$global:folders = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders" -Headers $headers
$global:ParentFolderID = $folders[0].value.parentFolderId

#Get Folder ID we are working with
foreach($folder in $folders.value)
{
    #Reset the Value
    $folderId = $NULL

    if($FolderName -eq $folder.displayName)
    {
        $folderId = $folder.id
        break
    }
}

Add-Content "$rootPath\progress.txt" "********** Finished Grabbing Contact Folder Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

##################### Check if our Contacts Folder exists. If it doesn't, create it. #####################

if($NULL -eq $folderId)
{
    Add-Content "$rootPath\progress.txt" "********** Creating Contact Folder $(Get-Date -Format "HH:mm ss")**********"
    Add-Content "$rootPath\progress.txt" ""

    $Start = Get-Date
    Write-Host "Creating Contacts Folder"

    Try
    {
        while($NULL = (Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId" -Headers $headers -Method get))
        {
            $NewContactFolder = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders" -Body $ContactsFolderBody -Headers $headers -Method post -ContentType 'application/json'
            sleep -Milliseconds 1
            $folderId = $($NewContactFolder.id)
        }
    }
    Catch
    {
        Out-Null
    }

    $End = Get-Date

    Write-Host "Contacts Folder created in $($Start - $End) seconds"
    Write-Host ""

    Add-Content "$rootPath\progress.txt" "********** Finished Creating Contact Folder $(Get-Date -Format "HH:mm ss")**********"
    Add-Content "$rootPath\progress.txt" ""
}

##################### Grab all of our User Information from AD #####################

Add-Content "$rootPath\progress.txt" "********** Grabbing AD User Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

$Start = Get-Date

$searcher=[adsisearcher]""
$searcher.Sort.PropertyName = "sn"
$searcher.Filter = "(&(objectcategory=person)(objectclass=user)(extensionAttribute2=custom)(|(mobile=*)(telephonenumber=*)))"

$colProplist = @(
    'givenname', 'extensionattribute2'
    'initials', 'mobile', 'telephonenumber'
    'sn', 'displayname', 'company'
    'title', 'mail', 'department'
    'thumbnailphoto', 'samaccountname'
)

$colPropList | & { process {
    $NULL = $searcher.PropertiesToLoad.Add($_)
}}

$End = Get-Date

Write-Host "User info took $($Start - $End) seconds"
Write-Host ""

Add-Content "$rootPath\progress.txt" "********** Finished Grabbing AD User Info $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

##################### Create our User Hashtable #####################

Add-Content "$rootPath\progress.txt" "********** Creating User Hashtable $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

Write-Host "Creating User Hashtable"
$Start = Get-Date

$users = $searcher.FindAll() | & { process {

    [pscustomobject]@{
        FirstName = [string]$_.properties.givenname
        MiddleName = [string]$_.properties.initials 
        LastName = [string]$_.properties.sn 
        DisplayName = [string]$_.properties.displayname 
        SamAccountName = [string]$_.properties.samaccountname 
        Email = [string]$_.properties.mail 
        Mobile = [string]$_.properties.mobile 
        TelephoneNumber = [string]$_.properties.telephonenumber 
        Title = [string]$_.properties.title 
        Dept = [string]$_.properties.department 
        Company = [string]$_.properties.company 
        Photo = [string]$_.properties.thumbnailphoto 
        ExtensionAttribute2 = [string]$_.properties.extensionattribute2
    }
}}

Write-Host "User Hashtable took $($Start - $End) seconds"
Write-Host ""

Add-Content "$rootPath\progress.txt" "********** Finished Creating User Hashtable $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

##################### Get Existing Contacts (Only if the Contacts Folder wasn't newly created )#####################

if($NULL -ne $folderId)
{
    Add-Content "$rootPath\progress.txt" "********** Grabbing Contact Info $(Get-Date -Format "HH:mm ss")**********"
    Add-Content "$rootPath\progress.txt" ""

    $Start = Get-Date

    $AllContacts = Invoke-RestMethod -Uri "https://graph.microsoft.com/v1.0/users/$UPN/contactFolders/$folderId/contacts?`$top=999&`$Orderby=Surname" -Headers $headers -Method Get

    $End = Get-Date

    Write-Host "Contact info took $($Start - $End) seconds"
    Write-Host ""

    Add-Content "$rootPath\progress.txt" "********** Finished Grabbing Contact Info $(Get-Date -Format "HH:mm ss")**********"
    Add-Content "$rootPath\progress.txt" ""

    ##################### Create our Contact Hashtable #####################

    Add-Content "$rootPath\progress.txt" "********** Creating Contact Hashtable $(Get-Date -Format "HH:mm ss")**********"
    Add-Content "$rootPath\progress.txt" ""

    Write-Host "Creating Contact Hashtable"
    $Start = Get-Date

    $Contacts = $AllContacts.value | & { process {

        [PSCustomObject]@{
            'FirstName' = [string]$_.givenName
            'MiddleName' = [string]$_.initials
            'LastName' = [string]$_.surname
            'DisplayName' = [string]$_.displayName
            'Email' = [string](($_.emailAddresses) | %{$_.Address})
            'Mobile' = [string]$_.mobilePhone
            'TelephoneNumber' = [string]$_.homePhones
            'Title' = [string]$_.jobTitle
            'Dept' = [string]$_.department
            'Company' = [string]$_.companyName
         }
     }}

    $End = Get-Date

    Write-Host "Contact HashTable took $($Start - $End) seconds"
    Write-Host ""

    Add-Content "$rootPath\progress.txt" "********** Finished Creating Contact Hashtable $(Get-Date -Format "HH:mm ss")**********"
    Add-Content "$rootPath\progress.txt" ""
}

##################### Start Comparing Data #####################

#Our Array of values we will be comparing
[array]$CompareValues = "FirstName","MiddleName","LastName","DisplayName","Email","Mobile","TelephoneNumber","Title","Dept","Company"

for($i=0; $i -lt $CompareValues.Count; $i++)
{
    #First let's create 2 variables that will hold the info we want
    $A = ($Users).($CompareValues[$i])
    $B = ($Contacts).($CompareValues[$i])

    ##################### Update Contacts #####################

    #Only Run if there are contacts; otherwise there is nothing for us to compare
    if(($NULL -ne $B))
    {
        #Displays all differences
        #$Differences = [string[]]([Linq.Enumerable]::Except([object[]]$a, [object[]]$b) + [Linq.Enumerable]::Except([object[]]$b, [object[]]$a))

        #Displays what accounts we need to import
        $NeedsToBeAdded = [string[]]([Linq.Enumerable]::Except([object[]]$a, [object[]]$b))

        #Displays what accounts we need to delete because they no longer exist
        $NeedsToBeDeleted = [string[]]([Linq.Enumerable]::Except([object[]]$b, [object[]]$a))
    }

    ##################### Import All Contacts #####################

    Else
    {
        Add-Content "$rootPath\progress.txt" "********** Importing All Contacts $(Get-Date -Format "HH:mm ss")**********"
        Add-Content "$rootPath\progress.txt" ""

        $Start = Get-Date

<#
        #---------------------------------------------
        . $rootPath\UpdateContacts\UpdateContacts.ps1

        # Set up runspace pool
        $RunspacePool = [runspacefactory]::CreateRunspacePool(1,10)
        $RunspacePool.Open()

        # Assign new jobs/runspaces to a variable
        $Runspaces = foreach ($User in $Users)
        {
            
            # Create new PowerShell instance to hold the code to execute, add arguments
            $PSInstance = [powershell]::Create().AddScript({
                $Users | ForEach{ UpdateContact($_) }
            }).AddParameter('$_')

            # Assing PowerShell instance to RunspacePool
            $PSInstance.RunspacePool = $RunspacePool

            # Start executing asynchronously, keep instance + IAsyncResult objects
            New-Object psobject -Property @{
                Instance = $PSInstance
                IAResult = $PSInstance.BeginInvoke()
                Argument = $User
            }
        }

        # Wait for the the runspace jobs to complete
        while($Runspaces |Where-Object{-not $_.IAResult.IsCompleted})
        {
            Start-Sleep -Milliseconds 500
        }

        # Collect the results
        $Results = $Runspaces |ForEach-Object {
            $Output = $_.Instance.EndInvoke($_.IAResult)
            New-Object psobject -Property @{
                User = $User
            }
        }

        #---------------------------------------------
#>
        Write-host "Importing Contacts. This could take several minutes."

        #There are no contacts, so let's import them
        
        #Path to our script that imports Contacts
        . $rootPath\UpdateContacts\UpdateContacts.ps1
        #$Users | & { process { UpdateContacts($_) } }

        #Start-ThreadJob -ScriptBlock { $Users | & { process { UpdateContacts($_) } } }
        Start-ThreadJob -ScriptBlock { $Users | ForEach{ UpdateContacts($_) } }
        Get-Job

        $End = Get-Date

        Write-Host "Contact Import took $($Start - $End) seconds"
        Write-Host ""

        Add-Content "$rootPath\progress.txt" "********** Finished Importing All Contacts $(Get-Date -Format "HH:mm ss")**********"
        Add-Content "$rootPath\progress.txt" ""
        break
    }
}

Add-Content "$rootPath\progress.txt" "********** Finished Script $(Get-Date -Format "HH:mm ss")**********"
Add-Content "$rootPath\progress.txt" ""

Answer 1

There is a bunch of code to go through so I'm gonna give you a blueprint of how you can achieve processing all users in $users using ThreadJob.

So, step by step, I'll try to add as much comments as I consider appropriate to guide you through the thought process.

I'm not sure what is the output of your function since I see an | Out-Null at the end of the Invoke-RestMethod. You would need to clarify on this.

# requires -Modules ThreadJob

# Load UpdateContacts function in memory
. "$rootPath\UpdateContacts\UpdateContacts.ps1"

# Save the function in a scriptBlock, we need this
# so we can pass this function in the scope of the ThreadJobs
$updateContacts = "function UpdateContacts { $function:updateContacts }"

# Define the Number of Threads we are going to use
# (Get-CimInstance win32_processor).NumberOfLogicalProcessors
# Can give you a good perspective as to how many Threads is safe to use.
$numberOfThreads = 10

# I'm assuming that $users is the array we want to process with
# the UpdateContacts function. Around 500 as you said in your question.
# Here I'm grouping the users in chunks so each running Job can process
# a chunk of users. Each chunk will contain around 50 users to process.
$groupSize = [math]::Ceiling($users.Count / $numberOfThreads)
$counter = [pscustomobject]@{ Value = 0 }
$chunks = $users | Group-Object -Property {
    [math]::Floor($counter.Value++ / $groupSize)
}

# Here is the magic
foreach($chunk in $chunks)
{
    # Capture this chunk of users in a variable
    $thisGroup = $chunk.Group
    
    # This is what we are running inside the scope
    # of our threadJob
    $scriptBlock = {

        # As in my comments, these variables don't exist inside here,
        # you need to pass them to these scope
        $UPN = $using:UPN
        $folderID = $using:folderId
        $headers = $using:headers
        $contactsBody = $using:contactsBody

        # First we need to define the function inside
        # this scope
        . ([scriptBlock]::Create($using:updateContacts))

        # Loop through each user
        foreach($user in $using:thisGroup)
        {
            UpdateContacts -User $user
        }
    
    } # EOF Job's ScriptBlock

    # ThrottleLimit is the number of Jobs that can run at the same time.
    # Be aware, a higher number of Jobs running does NOT mean that the
    # task will perform faster. This always depends on your CPU & Memory.
    # And, this case in particular, the number of requests your URI is able to handle
    Start-ThreadJob -ScriptBlock $scriptBlock -ThrottleLimit $numberOfThreads
}

# Now we should have 10 Jobs running at the same time, each Job
# is processing a chunk of 50 users aprox. (500 users / 10)
# Note: As in my previous comments, I see an Out-Null in your function
# not sure what is meant to return but in case, this is how you capture
# the output of all Jobs:
$result = Get-Job | Receive-Job -Wait

# Free up memory:
Get-Job | Remove-Job