Search code examples
powershellexceptionmemoryhigh-volume

Processing huge volume data file using powershell

I am trying to perform a replace operation on a data file which is 4GB. But I am not even able to read this file due to memory exception. The following command gives a memory error.

$edwfile = (Get-Content C:\Users\tomgeorg\Desktop\edw_ord_extr_3x_SIQP_20181021.182305\edw_ord_extr_3x_SIQP_20181021.182305.dat -Raw )

Is there any alternative commands or tricks to process huge file.

I want to run the following replace pattern on each line in the file.basically I want to remove all the unwanted special characters.

-replace  "[$([char]0x00)-$([char]0x09)$([char]0x0B)-$([char]0x1F)$([char]0x7F)-$([char]0xFF)]","?"

system details

enter image description here

Solution

  • Below is the sample solution with streams. It reads file line by line and then add updated line to a new file. 

    $reader = [System.IO.StreamReader]"C:\temp\OriginalFile.txt"
$writer = [System.IO.StreamWriter]"C:\temp\UpdatedFile.txt"

while (!$reader.EndOfStream) {

$writer.WriteLine(($reader.ReadLine() -replace '\|', ";"))

}

$reader.Close()
$writer.Close()