Search code examples
powershellpowershell-5.0

"OutOfMemoryException" when using Compress-Archive with 60Gb of files

I'm trying to compress IIS Log files (60GB total) using Compress-Archive, however I get the error:

"Exception of type 'System.OutOfMemoryException' was thrown."

$exclude = Get-ChildItem -Path . | Sort-Object -Descending | Select-Object -First 1
$ArchiveContents = Get-ChildItem -Path . -Exclude $exclude | Sort-Object -Descending
Compress-Archive -Path $ArchiveContents -DestinationPath .\W3SVC2.zip -Force

I've already adjusted MaxMemoryPerShellMB to 2048MB and restarted the WinRM service.

RAM Memory consumption exceeds 6GB when the command is executed.

Solution

  • As suggested by Matt, I would recommend archiving it into sets of files. Something like:

    For($i=0;$i -le $ArchiveContents.Count;$i=$i+10){
    Compress-Archive -Path $ArchiveContents[$i..($i+9)] -DestinationPath .\W3SVC2-$i.zip -Force
}

    That way you're only working with 10 files at a time. You may want to adjust the number of files, depending on how many you have, and how large they are, ie: If you have 300 files at 200MB each then 10 at a time may be good, whereas if you have 3000 files at 20MB each you may want to increase that to 50 or even 100.

    Edit: I just looked, and Compress-Archive supports adding files to an archive by specifying the -Update parameter. You should be able to do the same thing as above, slightly modified, to make 1 large archive.

    For($i=0;$i -le $ArchiveContents.Count;$i=$i+10){
    Compress-Archive -Path $ArchiveContents[$i..($i+9)] -DestinationPath .\W3SVC2.zip -Update
}

    That should just add 10 files at a time to the target archive, rather than trying to add all of the files at once.