I am currently trying to run a parallelized RStan job on a computing cluster in R. I am doing this by specifying the following two options:
options(mc.cores = parallel::detectCores())
rstan_options(auto_write = TRUE)
The above allocates 48 cores available and the total RAM I have is 180 GB. I always thought in theory that more cores and more RAM was better. I am running very long jobs and I am getting insufficient memory errors in my cluster. I am wondering if I am perhaps not giving enough memory to each core. Is it possible that the 48 cores each are splitting the 180 GB and each core is then maxed out?
If I were to use the 180 GB of RAM and instead had 3 cores, would this get around memory errors? Or is it no matter how many cores I have, the total memory will always be used up at some point if its a long job? Thanks!
RStan is only going to utilize as many cores as there are chains (by default 4). And if you are using a shell on a Linux cluster, it will fork rather than making copies of the data. So, if you have idle cores, you are better off utilizing them to parallelize computations within a Stan program using the map_rect function, if possible.
But all of that is probably unrelated to your memory problems. It should not require even 1 GB in most cases.