Using Powershell to replace multiple strings in multiple files & folders

Question

I have a list of strings in a CSV file. The format is:

OldValue,NewValue
223134,875621
321321,876330
....

and the file contains a few hundred rows (each OldValue is unique). I need to process changes over a number of text files in a number of folders & subfolders. My best guess of the number of folders, files, and lines of text are - 15 folders, around 150 text files in each folder, with approximately 65,000 lines of text in each folder (between 400-500 lines per text file).

I will make 2 passes at the data, unless I can do it in one. First pass is to generate a text file I will use as a check list to review my changes. Second pass is to actually make the change in the file. Also, I only want to change the text files where the string occurs (not every file).

I'm using the following Powershell script to go through the files & produce a list of the changes needed. The script runs, but is beyond slow. I haven't worked on the replace logic yet, but I assume it will be similar to what I've got.

# replace a string in a file with powershell
[reflection.assembly]::loadwithpartialname("Microsoft.VisualBasic") | Out-Null

Function Search {
  # Parameters $Path and $SearchString
  param ([Parameter(Mandatory=$true, ValueFromPipeline = $true)][string]$Path,
  [Parameter(Mandatory=$true)][string]$SearchString
  )
  try {
    #.NET FindInFiles Method to Look for file

    [Microsoft.VisualBasic.FileIO.FileSystem]::GetFiles(
    $Path,
    [Microsoft.VisualBasic.FileIO.SearchOption]::SearchAllSubDirectories,
    $SearchString
    )
  } catch { $_ }

}


if (Test-Path "C:\Work\ListofAllFilenamesToSearch.txt") {  # if file exists
    Remove-Item "C:\Work\ListofAllFilenamesToSearch.txt"
    }
if (Test-Path "C:\Work\FilesThatNeedToBeChanged.txt") {  # if file exists
    Remove-Item "C:\Work\FilesThatNeedToBeChanged.txt"
    }

$filefolder1 = "C:\TestFolder\WorkFiles"
$ftype = "*.txt"
$filenames1 = Search $filefolder1 $ftype

$filenames1 | Out-File "C:\Work\ListofAllFilenamesToSearch.txt" -Width 2000

if (Test-Path "C:\Work\FilesThatNeedToBeChanged.txt") {  # if file exists
    Remove-Item "C:\Work\FilesThatNeedToBeChanged.txt"
    }

(Get-Content "C:\Work\NumberXrefList.CSV" |where {$_.readcount -gt 1}) | foreach{
    $OldFieldValue, $NewFieldValue = $_.Split("|")
    $filenamelist = (Get-Content "C:\Work\ListofAllFilenamesToSearch.txt" -ReadCount 5) #| 
    foreach ($j in $filenamelist) {
    #$testvar = (Get-Content $j )
    #$testvar = (Get-Content $j -ReadCount 100)
    $testvar = (Get-Content $j -Delimiter "\n")
            Foreach ($i in $testvar)
            {
            if ($i -imatch $OldFieldValue) {
                $j + "|" + $OldFieldValue + "|" + $NewFieldValue | Out-File "C:\Work\FilesThatNeedToBeChanged.txt" -Width 2000 -Append
                }
            }
    }
}

$FileFolder = (Get-Content "C:\Work\FilesThatNeedToBeChanged.txt" -ReadCount 5)

Get-ChildItem $FileFolder -Recurse |
select -ExpandProperty fullname |
foreach {
   if (Select-String -Path $_  -SimpleMatch $OldFieldValue -Debug -Quiet) {
      (Get-Content $_) |
      ForEach-Object {$_ -replace $OldFieldValue, $NewFieldValue }|
      Set-Content $_ -WhatIf
    }
}

In the code above, I've tried several things with Get-Content - default, with -ReadCount, and -Delimiter - in an attempt to avoid an out of memory error.

The only thing I have control over is the length of the old & new replacement strings file. Is there a way to do this in Powershell? Is there a better option/solution? I'm running Windows 7, Powershell version 3.0.

Answer 1

Your main problem is that you're reading the file over and over again to change each of the terms. You need to invert the looping of the replace terms and looping of the files. Also, pre-load the csv. Something like:

$filefolder1 = "C:\TestFolder\WorkFiles"
$ftype = "*.txt"
$filenames = gci -Path $filefolder1 -Filter $ftype -Recurse

$replaceValues = Import-Csv -Path "C:\Work\NumberXrefList.CSV"

foreach ($file in $filenames) {
    $contents = Get-Content -Path $file

    foreach ($replaceValue in $replaceValues) {    
        $contents = $contents -replace $replaceValue.OldValue, $replaceValue.NewValue
    }

    Copy-Item $file "$file.old"
    Set-Content -Path $file -Value $contents
}