Get-ChildItem in an enormous directory and RAM usage

Question

I have created a PS script on a domain controller (SRV2012R2).

The script checks every shared folder (mapped drive) to see if there are any files present larger than 2GB:

foreach($dir in $Dirs)
{
    $files = Get-ChildItem $dir -Recurse 

    foreach ($item in $files)
    {

       #Check if $item.Size is greater than 2GB

    }
}

I have the following problem:

The shares are pretty filled with over 800GB of (sub)folders, files and most of them are just normal documents.

Whenever I run my script, I see that the CPU+RAM consumes enormous amounts while running my script (after 5 minutes into the Get-Childitem-line, the RAM has already reached >4GB).

My question is, why does Get-ChildItem need so many resources? What alternative can I use? Because I haven't manage to run my script succesfully.

I have seen that I can use | SELECT fullname, length after my Get-ChildItem-clause as an improvement, but this hasn't helped me at all (query still consumes enormous RAM).

Is there anything I can do in order that I can loop through the directories without much resistance from the machine resources?

Answer 1

Instead of saving every single file to a variable, use the pipeline to filter out the ones you don't need (ie. those smaller than 2GB):

foreach($dir in $Dirs)
{
    $bigFiles = Get-ChildItem $dir -Recurse |Where Length -gt 2GB
}

If you need to process or analyze these big files further, I'd suggest you extend that pipeline with ForEach-Object:

foreach($dir in $Dirs)
{
    Get-ChildItem $dir -Recurse |Where Length -gt 2GB |ForEach-Object {
      # do whatever else you must
      # $_ contains the current file
    }
}