Powershell limiting BITS transfers to 4 at a time

Question

I'm working on a script to transfer Office 365 ProPlus binaries to 12 separate file shares using BITS. I want to also display progress of these transfers on screen. My issue is, I want to limit the number of simultaneous BITS transfers to no more than 4 at a time. As one job completes, I then want to start the next job in queue and continue to receive progress until all jobs complete.

This is what I have so far. I first start with this function to create all my BITS jobs for all network locations in a suspended state.

function Start-BinaryTransfer
{
[CmdletBinding()]
param
(
    [array]$DistrictsToUpdate
)

$Source = "$BaseSource\$UpdateChannel"

if ("All" -in $DistrictsToUpdate.Split(','))
{
    foreach ($Destination in $ReposList)
    {
        Copy-Item -Path $Source -Destination $($Destination.Location + '\') -Filter { $_.PSisContainer } -Recurse -ErrorAction SilentlyContinue
        Get-ChildItem -Path $Source -Recurse | Where-Object{ $_.PSisContainer } | ForEach-Object {
            $spath = $_.FullName.Remove(0, $Source.Length + 1)
            $BITSJobs += Start-BitsTransfer -Source $Source\$spath\*.* `
                                            -Destination "$($Destination.Location)\$UpdateChannel\$spath" `
                                            -DisplayName "$($Destination.District) File Transfer" `
                                            -Description "Transferring from [$Source] to [$($Destination.Location)\$UpdateChannel]" `
                                            -Suspended
        }
    }
}

Once all jobs have been created, I then try to use this While loop to start 4 jobs at a time, and display progress as I go. Unfortunately, the actual behavior is that it will attempt to start all 12 jobs at once, which then bogs down network resources.

While (Get-BitsTransfer | Where JobState -EQ "Suspended")
{
Get-BitsTransfer | Where JobState -EQ "Suspended" | ForEach-Object {
    for ($JobsCount = 0; $JobsCount -le 4; $JobsCount++)
    {
        if ($JobsCount -lt 4)
        {
            Resume-BitsTransfer -BitsJob $_ -Asynchronous
            Get-BitsTransfer | Where JobState -EQ "Transferring" | ForEach-Object {
                Write-Progress `
                               -Id $([math]::Abs($_.DisplayName.GetHashCode())) `
                               -Activity "$($_.DisplayName)" `
                               -Status "$($_.Description)" `
                               -CurrentOperation "$([math]::Floor($_.BytesTransferred / $_.BytesTotal * 100)) % Complete" `
                               -PercentComplete $([math]::Floor($_.BytesTransferred / $_.BytesTotal * 100))
            }
        }
      }
    }

    if (Get-BitsTransfer | Where JobState -EQ "Transferred")
    {
        Get-BitsTransfer | Where JobState -EQ "Transferred" | Complete- BitsTransfer
        $JobsCount--
    }
}

Answer 1

With the for-loop, you are resetting $JobsCount to 0 every run

for ($JobsCount = 0; $JobsCount -le 4; $JobsCount++)

Something like this should work (with minimal amount of changes to your current code):

$JobsCount = 0
While (Get-BitsTransfer | Where JobState -EQ "Suspended")
{
Get-BitsTransfer | Where JobState -EQ "Suspended" | ForEach-Object {
    # remove for-loop
        if ($JobsCount -lt 4)
        {
            $JobsCount++
            ...

You also have to modify your while-loop, currently it will exit when there're no more jobs with JobState "Suspended", but you still have to wait for jobs with JobState "Transferring" and complete them when they are "Transferred". Suggestion: While ((Get-BitsTransfer) -ne $null) or even more simple While (Get-BitsTransfer)