How to optimize Powershell script with Get-ChildItem consuming all RAM

Question

I have this script which parses all shares on a file server to gather information on share size, ACLs, and count of files and folders. The script works great on smaller file servers but on hosts with large shares it consumes all RAM and crashes the host, I can't seem to figure out how to optimize the script during the Get-ChildItem portion to not consume all RAM.

I found a few articles which mentioned to use a foreach loop and pipe out what I need. I am a Powershell beginner, I can't figure out how to get it to work like that. What can I try next?

$ScopeName     = Read-Host "Enter scope name to gather data on"
$SavePath      = Read-Host "Path to save results and log to"
$SaveCSVPath   = "$SavePath\ShareData.csv"
$TranscriptLog = "$SavePath\Transcript.log"

Write-Host
Start-Transcript -Path $TranscriptLog

$StartTime = Get-Date
$Start     = $StartTime | Select-Object -ExpandProperty DateTime

$Exclusions = {$_.Description -ne "Remote Admin" -and $_.Description -ne "Default Share" -and $_.Description -ne "Remote IPC" }
$FileShares = Get-SmbShare -ScopeName $ScopeName | Where-Object $Exclusions
$Count      = $FileShares.Count
Write-Host
Write-Host "Gathering data for $Count shares" -ForegroundColor Green
Write-Host
Write-Host "Results will be saved to $SaveCSVPath" -ForegroundColor Green
Write-Host

ForEach ($FileShare in $FileShares)
{
    $ShareName = $FileShare.Name
    $Path      = $Fileshare.Path

    Write-Host "Working on: $ShareName - $Path" -ForegroundColor Yellow
    
    $GetObjectInfo = Get-Childitem -Path $Path -Recurse -Force -ErrorAction SilentlyContinue

    $ObjSize = $GetObjectInfo | Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue

    $ObjectSizeMB = "{0:N2}" -f ($ObjSize.Sum / 1MB)
    $ObjectSizeGB = "{0:N2}" -f ($ObjSize.Sum / 1GB)
    $ObjectSizeTB = "{0:N2}" -f ($ObjSize.Sum / 1TB)

    $NumFiles   = ($GetObjectInfo | Where-Object {-not $_.PSIsContainer}).Count
    $NumFolders = ($GetObjectInfo | Where-Object {$_.PSIsContainer}).Count
    
    $ACL            = Get-Acl -Path $Path
    $LastAccessTime = Get-ItemProperty $Path | Select-Object -ExpandProperty LastAccessTime
    $LastWriteTime  = Get-ItemProperty $Path | Select-Object -ExpandProperty LastWriteTime

    $Table = [PSCustomObject]@{
        'ScopeName'          = $FileShare.ScopeName
        'Sharename'          = $ShareName
        'SharePath'          = $Path
        'Owner'              = $ACL.Owner
        'Permissions'        = $ACL.AccessToString
        'LastAccess'         = $LastAccessTime
        'LastWrite'          = $LastWriteTime
        'Size (MB)'          = $ObjectSizeMB
        'Size (GB)'          = $ObjectSizeGB
        'Size (TB)'          = $ObjectSizeTB
        'Total File Count'   = $NumFiles
        'Total Folder Count' = $NumFolders
        'Total Item Count'   = $GetObjectInfo.Count
    }

    $Table | Export-CSV -Path $SaveCSVPath -Append -NoTypeInformation 
}

$EndTime = Get-Date
$End     = $EndTime | Select-Object -ExpandProperty DateTime

Write-Host
Write-Host "Script start time: $Start" -ForegroundColor Green
Write-Host "Script end time: $End" -ForegroundColor Green

Write-Host
$ElapsedTime = $(($EndTime-$StartTime))
Write-Host "Elapsed time: $($ElapsedTime.Days) Days $($ElapsedTime.Hours) Hours $($ElapsedTime.Minutes) Minutes $($ElapsedTime.Seconds) Seconds $($ElapsedTime.MilliSeconds) Milliseconds" -ForegroundColor Cyan

Write-Host
Write-Host "Results saved to $SaveCSVPath" -ForegroundColor Green

Write-Host
Write-Host "Transcript saved to $TranscriptLog" -ForegroundColor Green

Write-Host
Stop-Transcript

Answer 1

To correctly use the PowerShell pipeline (and preserve memory as each item is streamed separately), use the PowerShell ForEach-Object cmdlet (unlike the ForEach statement) and avoid assigning the pipeline to a variable (as you doing with $FileShares = ...) and don't use parenthesis ((...)) arround the the pipeline:

Get-SmbShare -ScopeName $ScopeName | Where-Object $Exclusions | ForEach-Object {

And replace all $FileShare variables in your loop with the current item: $_ variable (e.g. $FileShare.Name → $_.Name).

For the Get-Childitem part you might do the same thing (stream! meaning: use the mighty PowerShell pipeline rather than piling everything up in $GetObjectInfo):

$ObjSize = Get-Childitem -Path $Path -Recurse -Force -ErrorAction SilentlyContinue |
    Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue

As an aside; you might simplify your 3 size properties to a single smarter size property, see: How to convert value to KB, MB, or GB depending on digit placeholders?

addition
"But isn't putting everything into $ObjSize just swapping one variable for another?"
No it is not, think of the PowerShell pipeline as an assembly line. In this case, at the first station you take each single file information and pass it to the next (last) station where you just sum the length property and the current (file) object disposed.
Where in your question example, you read the information of all files in once and store it into $GetObjectInfo and than go to the whole list to just use (add) the length property of the (quiet heavy) PowerShell file objects.

But why don't you try it?:

Open a new PowerShell session and run:

$Path = '.'
$GetObjectInfo = Get-Childitem -Path $Path -Recurse -Force -ErrorAction SilentlyContinue
$ObjSize = $GetObjectInfo | Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue
Get-Process -ID $PID

Now, open a new session again and use the PowerShell pipeline:

$Path = '.'
$ObjSize = Get-Childitem -Path $Path -Recurse -Force -ErrorAction SilentlyContinue |
           Measure-Object -Property Length -Sum -ErrorAction SilentlyContinue
Get-Process -ID $PID

Notice the difference in memory usage (WS(M)).