Search code examples
rmergesparse-matrixseurat

Cholmod error 'out of memory' : Merging Seurat Objects

I am trying to merge Seurat class objects that contain transcriptome count data (sparse matrix). I am relatively new to R, so any help/solutions is appreciated. I have added a screenshot of the data I'm working with.

**General Info:**
-------------

> memory.size(max = TRUE)
[1] 2533.94
R version 4.0.3 (2020-10-10)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 10 x64 (build 19041)

attached base packages:
[1] stats     graphics  grDevices utils    
[5] datasets  methods   base     

other attached packages:
[1] RSQLite_2.2.3 Seurat_3.2.3

I am not sure if my storage is the issue or if I should split the function in two.

 options(stringsAsFactors = F)
setwd("C:/Users/Amara/OneDrive - Virginia Tech/XieLab/ZebraFish_Project/zf_brain-master/data")
folders <- list.files("C:/Users/Amara/OneDrive - Virginia Tech/XieLab/ZebraFish_Project/zf_brain-master/data")
library(Seurat)
library(dplyr)
zfbrainList = lapply(folders,function(folder){ 
  CreateSeuratObject(counts = Read10X(folder), 
                     project = folder )
})
zfbrain.combined <- merge(zfbrainList[[1]], 
                 y = c(zfbrainList[[2]],zfbrainList[[3]],zfbrainList[[4]],zfbrainList[[5]],
                       zfbrainList[[6]],zfbrainList[[7]],zfbrainList[[8]],zfbrainList[[9]],
                       zfbrainList[[10]],zfbrainList[[11]],zfbrainList[[12]],zfbrainList[[13]],
                       zfbrainList[[14]],zfbrainList[[15]]), 
                 add.cell.ids = folders, 
                 project = "zebrafish")

Error in .cbind2Csp(x, y) : Cholmod error 'out of memory' at file ../Core/cholmod_memory.c, line 147

Data folder

zfbrainlist

Solution

  • The machine used to process the data in the original question has a 64-bit Windows operating system running a 32-bit version of R. The result from memory.size() shows that approximately 2.4Gb of RAM is available to the malloc() function used by R. The 32-bit version of R on Windows can access a maximum of slightly less than 4Gb of RAM when running on 64-bit Windows, per the help for memory.size().

    enter image description here

    Memory Limits in R tells us that in 32-bit R on Windows it is usually not possible to allocate a single vector of 2Gb in size due to the fact that windows consumes some memory in the middle of the 2 Gb address space.

    enter image description here

    Once we load the data from the question, the zfbrainList object consumes about 1.2Gb of RAM.

    options(stringsAsFactors = F)
folders <- list.files("./data/zebraFishData",full.names = TRUE)
library(Seurat)
library(dplyr)
zfbrainList = lapply(folders,function(folder){ 
     CreateSeuratObject(counts = Read10X(folder), 
                        project = folder )
})
format(object.size(zfbrainList),units = "Gb")

    ...and the result:

    > format(object.size(zfbrainList),units = "Gb")
[1] "1.2 Gb"

    At this point, the code attempts to merge the objects from the list into a single object.

    zfbrain.combined <- merge(zfbrainList[[1]], 
                          y = c(zfbrainList[[2]],zfbrainList[[3]],zfbrainList[[4]],zfbrainList[[5]],
                                zfbrainList[[6]],zfbrainList[[7]],zfbrainList[[8]],zfbrainList[[9]],
                                zfbrainList[[10]],zfbrainList[[11]],zfbrainList[[12]],zfbrainList[[13]],
                                zfbrainList[[14]],zfbrainList[[15]]), 
                          add.cell.ids = folders, 
                          project = "zebrafish")

    When we calculate the size of the resulting zfbrain.combined object, we find that it is also about 1.2Gb in size, which exceeds the RAM available to R on the original poster's machine.

    format(object.size(zfbrain.combined),units = "Gb")

> format(object.size(zfbrain.combined),units = "Gb")
[1] "1.2 Gb"

    Since the zfbrainList must be in RAM while zfbrain.combined is being created, it is not possible to execute the merge as coded above in an instance of R that has only 2.4Gb of RAM accessible because the RAM consumed by both zfbrainList and zfbrain.combined is between 2.4 - 2.5Gb, exclusive of other RAM needed by R to run.

    Solution: use the 64-bit version of R

    Since most Windows-based machines have at least 4Gb of RAM, and the amount of RAM reported by memory.size() was 2.4Gb, it's likely there is at least 4Gb of RAM on the machine. The machine used in the original post already had 64-bit Windows installed, so we can enable R to access more memory by installing and running the 64-bit version of R.

    On a Windows-based machine with 8Gb RAM, 32-bit R reports the following for memory.size() and memory.limit().

    enter image description here

    Interestingly, R reports 25.25 for memory.size() because 1Mb is rounded down to 0.01 per the help documentation, but memory.limit() provides a number between 0 and 4095 (also per the documentation). On our test machine it reports 3583, about 3.5Gb of RAM.

    When we run these functions in 64-bit R on the same machine, memory.size() reports 34.25, which means that malloc() will allocate a single object as large as 3.3Gb, and memory.limit() reports that R can access a total of 8Gb of RAM, the total amount that is installed on this particular machine.

    enter image description here

    Testing the solution

    When I run the code in a 32-bit R 4.0.3 session on 64-bit Windows, I am able to replicate the out of memory error.

    enter image description here

    When I run the code in the 64-bit version of R, it runs to completion, and I am able to calculate the size of the resulting zfbrain.combined object.

    enter image description here