Can't Get Powershell to assign an array over a certain size

Question

So, I'm trying to create a powershell script that "benchmarks" the speed of GPG using various size files. The part in specific that is causing me trouble is this:

1 $temppath="$($env:Temp)\"
...
4 Measure-Command {$bytes= 10000MB
5 [System.Security.Cryptography.RNGCryptoServiceProvider]$rng = New- Object.System.Security.Cryptography.RNGCryptoServiceProvider
6 $rndbytes = New-Object byte[] $bytes
7 [System.Io.File]::WriteAllBytes("$($temppath)\test.txt", $rndbytes)} | Select-Object -OutVariable gentime | Out-Null; 

When $bytes >~ 1.5GB (I use 10000MB to test it), I get an error that, If I'm understanding what's going on correctly, corresponds to the fact that the byte array is indexed by a variable of type int32. The specific error is this:

New-Object : Cannot convert argument "0", with value: "10485760000", for "Byte[]" to type "System.Int32": "Cannot convert value "10485760000" to type "System.Int32". Error: "Value was either too large or too small for an Int32.""
At line:6 char:13
+ $rndbytes = New-Object byte[] $bytes
    + CategoryInfo          : InvalidOperation: (:) [New-Object], MethodException
    + FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand

Is there a way to force powershell to index $rndbytes, when the object is created, with an index of type int64 or uint64? If that's not the issue, I'd love to be shown and have what's exactly going wrong explained.

Do note, if you use, for example $bytes=1000MB (or write $bytes=1GB), the code works perfectly. I suspect I'm hitting a wall at $bytes.index > 2^31 -1 or so.

Thanks for the help~!!

Answer 1

As mentioned in the comments, arrays in .NET cannot exceed 2^31-1 items in length, because the index value used for arrays is [int] - the max value of which is 2^31-1.

It's not really a problem, because buffering 2 billion items in memory is terribly inefficient anyways - if you want 10GB of random data, you generate and write it to disk in chunks instead:

# Set file size
$intendedSize = 10GB

# Create output buffer
$buffer = [byte[]]::new(4KB)

# Create RNG
$rng = [System.Security.Cryptography.RNGCryptoServiceProvider]::Create()

# Create temp file, open writable filestrem
$tempFile = [System.IO.Path]::GetTempFileName() |Get-Item
$fileStream = $tempFile.OpenWrite()

do {
    # Fill buffer, write to disk, rinse-repeat
    $rng.GetBytes($buffer)
    $fileStream.Write($buffer, 0, $buffer.Length)
} while($fileStream.Length -lt $intendedSize)

# Dispose if filestream + RNG
$fileStream.Dispose()
$rng.Dispose()

4KB is ususally a good general buffer size on Windows, since 4KB is the default cluster size on NTFS volumes and most modern CPUs have L1 cache sizes divisible by 4KB