Powershell: The specified path, or filename, or both, are too long - Directory Excludes

Question

We’ve been running this script fine previously, but recently we’ve been getting some issues (See below error) with our config transform step from our TFS Build.

$serviceFiles = Get-ChildItem $localWorkspace -Recurse | where {$_.Extension -eq ".exe"}

We recently switched to using Gulp to compile our CSS and JS which has given us a “node_modules” folder. It looks like it’s trying to crawl these folders and actually gets to the directory length limit. I’ve tried various suggestions I’ve found from googling and other related questions on SO, but none seem to be working for me, it’s still hitting the same error (and I assume isn’t actually excluding these folders)

This is an example of a modified version I've tried using to exclude the folders

$excludedDirectories = @("*node_modules*", "*packages*", "*Common*");
Write-Verbose $localWorkspace

# get services for config types (service configs named same as assmebly .exe)
$serviceFiles = Get-ChildItem $localWorkspace -Recurse -Exclude $excludedDirectories | ?{$_.Extension -eq ".exe" -and $_.FullName -notmatch '*node_modules*' }

I've tried some variations on this based on other SO questions and answers, but the solution evades me. I've read from a few sources that the -Exclude doesn't work for a lot of people in most situations, so i tried the solution of a where clause to exclude a folder (I want to exclude multiple, but I tried just node_modules to see if I could get it past that, the other folders aren't too deep)

I want to exclude the node_modules directory, along with a couple of others that don't need to be checked for a transform. Any help would be greatly appreciated, thank you.

Answer 1

Unfortunately, the -Exclude parameter won't exclude the directories before enumerating all the files. So it still errors out. You need to exclude them earlier than that.

If those directories only exist at the top level, you can enumerate the top level directories, exclude the ones you don't want, and then examine the contents inside the remaining directories. Like this:

$excludedDirectories = @("node_modules", "packages", "Common")

$serviceFiles = Get-ChildItem $localWorkspace -Exclude $excludedDirectories | % { Get-ChildItem $_ -Recurse } | ? {$_.Extension -eq ".exe" }

Also, you can make this a bit simpler by using -Include:

$serviceFiles = Get-ChildItem $localWorkspace -Exclude $excludedDirectories | % { Get-ChildItem $_ -Recurse -Include "*.exe" }

Notice what I did. I removed the top level -Recurse and filtered out those directories. Then I used -Recurse on the remaining children of the top-most parent, giving us the files we're looking for.

If the directories you need to filter out appear deep in the hierarchy or at multiple levels, you'll have to write your own recursive traversal function:

function Get-ChildItemRecursiveExclude(
    [Parameter(Mandatory=$true)][string[]]$Path,
    [Parameter(Mandatory=$true)][string[]]$ExcludedDirNames
) {
    $immediateChildren = Get-ChildItem $Path -Exclude $ExcludedDirNames
    foreach ($c in $immediateChildren) {
        # Uncaptured output is returned
        $c
        if (Test-Path $c -PathType Container) {
            Get-ChildItemRecursiveExclude $c $ExcludedDirNames
        }
    }
}

$serviceFiles = Get-ChildItemRecursiveExclude $localWorkspace @("node_modules", "packages", "Common") | ? { $_.Extension -eq ".exe" }

Overall, the basic idea is that you have to keep PowerShell from traversing down into node_modules in the first place. npm creates very deep hierarchies that surpass the old limits on path lengths. (I'm not really clear why .NET still enforces them, but it does even though some of the underlying Windows API no longer does. for example, robocopy and several third party runtimes, like Node, don't bother with them.)