Search code examples
rspssdata-processingspss-modeler

R scripting in SPSS Modeler 16: change default "rowCount=1000" for modelerData

When applying R transform Field operation node in SPSS Modeler, for every script, the system will automatically add the following code on the top of my own script to interface with the R Add-on:

  while(ibmspsscfdata.HasMoreData()){
  modelerDataModel <- ibmspsscfdatamodel.GetDataModel()
  modelerData <-  ibmspsscfdata.GetData(rowCount=1000,missing=NA,rDate="None",logicalFields=FALSE)

Please note "rowCount=1000". When I process a table with >1000 rows (which is very normal), errors occur.

Looking for a way to change the default setting or any way to help to process table >1000 rows!

Solution

  • I've tried to add this at the beggining of my code and it works just fine:

    while(ibmspsscfdata.HasMoreData())
{
modelerData <-rbind(modelerData,ibmspsscfdata.GetData(rowCount=1000,missing=NA,rDate="None",logicalFields=FALSE))
}

    Note that you will consume a lot of memory with "big data" and parameters of .GetData() function should be set accordingly to "Read Data Options" in node setting.