rExcel getarraytovba size limit

Question

I've got some R code that returns a matrix that is 12 columns wide with over 1M rows. When I try to use getarraytovba to return that matrix into a VBA variant, it fails. When I say it fails I mean that it runs the code without generating any errors but the VBA variable will be empty. If I shrink the R matrix to below 5000 rows then VBA will capture the variable. If it is between 5000 and 20000 (ball park) then sometimes it will work and sometimes it won't. My system has 16GBs of ram and is only 40% utilized when I'm attempting to move the data to VBA. The memory usage doesn't seem to change as I have task manager open as I'm running the code.

I've googled the subject and the only answer I've found is that it is limited by physical memory but since I have nearly 10GB of free memory I think there is more to it than just that. Can anyone help me shed light on why getarraytovba is so limiting?

Answer 1

I wrote the following in VBA to address the shortcoming...

Public Function returnresults()
Dim lResultsize As Long
Dim sBigblock As Variant
Dim lLow As Long
Dim lHigh As Long
Dim vTemp As Variant
Dim i As Long
Dim j As Long
Dim lBigrow As Long
Dim lFullresults As Long
rinterface.RRun "abc<-length(vbaget[,1])"
lFullresults = rinterface.GetRExpressionValueToVBA("abc")
lResultsize = lFullresults
If lResultsize > 1048575 Then
MsgBox "Results exceed 1,048,575 rows.  Excess will be dropped."
lResultsize = 1048575
End If
sBigblock = ThisWorkbook.Sheets("results").Range("a2:m" & lResultsize + 1)
lHigh = lResultsize
lLow = 1
If lResultsize > 3000 Then lHigh = 3000

lBigrow = 1
Do While lHigh <= lResultsize And lLow < lHigh
    rinterface.RRun "temp<-vbaget[" & lLow & ":" & lHigh & ",]"
    vTemp = rinterface.GetArrayToVBA("temp")
        For i = 0 To UBound(vTemp, 1)
            For j = 1 To 13 'This is number of columns in array it could be dynamic
                sBigblock(lBigrow, j) = vTemp(i, j - 1)
            Next j
            lBigrow = lBigrow + 1
        Next i
    lLow = lHigh + 1
    lHigh = lLow + 2999
    If lHigh > lResultsize Then lHigh = lResultsize
Loop

ThisWorkbook.Sheets("results").Range("a2:m" & lResultsize + 1) = sBigblock
End Function