Search code examples
daskopenblas

"Too many memory regions" error with Dask

When using Dask with Dask array I suddenly get the following error and my kernel that dies / restarts.

The console says: 

BLAS : Program is Terminated. Because you tried to allocate too many memory regions

I'm using Anaconda on mac with OpenBLAS. Any idea?

Note: this question was given to me through other means, I'm repeating it here for future reference

Solution

  • Short Answer

    Dask and OpenBLAS are both trying to parallelize at the same time. This triggers a check in OpenBLAS which causes a hard failure. A simple way to avoid this is to set the following environment variable:

    export OMP_NUM_THREADS=1

    Long Answer

    Dask is calling OpenBLAS functions (through NumPy) many times in parallel. OpenBLAS in turn starts up many threads to operate in parallel. So if you had four cores then you might end up with sixteen threads all trying to do work. This can cause poor performance. OpenBLAS intelligently identifies this situation and warns you about it. Unfortunately its mechanism to warn you is to fail hard.

    Setting OMP_NUM_THREADS=1 tells OpenBLAS to only use one thread at a time. This will help avoid contention, but may result in slower performance when not parallelizing with some other library like Dask or Joblib.