Search code examples
pythoncachingpermissionscupy

changing cache dir due to permission problems


I try to run cupy using SLURM on a HPC at my university. I get the following error:

PermissionError: [Errno 13] Permission denied: '/home/username'

I think this happens because the compute nodes do not have the permission to access /home. Since the whole error says

    Traceback (most recent call last):
  File "/shared-storage/username/tests/cuml_test.py", line 12, in <module>
    dbscan_float.fit(gdf_float)
  File "cuml/cluster/dbscan.pyx", line 219, in cuml.cluster.dbscan.DBSCAN.fit
  File "/shared-storage/username/miniconda/lib/python3.6/site-packages/cuml/ut
ils/input_utils.py", line 139, in input_to_cuml_array
    X_m = CumlArray(data=cuml.utils.numba_utils.row_matrix(X))
  File "/shared-storage/username/miniconda/lib/python3.6/site-packages/cuml/ut
ils/memory_utils.py", line 55, in cupy_rmm_wrapper
    return func(*args, **kwargs)
  File "/shared-storage/username/miniconda/lib/python3.6/site-packages/cuml/ut
ils/numba_utils.py", line 36, in row_matrix
    col_major = df.as_gpu_matrix(order='F')
  File "/shared-storage/username/miniconda/lib/python3.6/site-packages/cudf/co
re/dataframe.py", line 1926, in as_gpu_matrix
    matrix[:, colidx] = dense
  File "cupy/core/core.pyx", line 1248, in cupy.core.core.ndarray.__setitem__
  File "cupy/core/_routines_indexing.pyx", line 49, in cupy.core._routines_index
ing._ndarray_setitem
  File "cupy/core/_routines_indexing.pyx", line 800, in cupy.core._routines_inde
xing._scatter_op
  File "cupy/core/core.pyx", line 517, in cupy.core.core.ndarray.fill
  File "cupy/core/_kernel.pyx", line 648, in cupy.core._kernel.ElementwiseKernel
.__call__
  File "cupy/core/_kernel.pyx", line 678, in cupy.core._kernel.ElementwiseKernel
._get_elementwise_kernel
  File "cupy/core/_kernel.pyx", line 478, in cupy.core._kernel._get_elementwise_
kernel
  File "cupy/core/_kernel.pyx", line 41, in cupy.core._kernel._get_simple_elemen
twise_kernel
  File "cupy/core/_kernel.pyx", line 61, in cupy.core._kernel._get_simple_elemen
twise_kernel
  File "cupy/core/carray.pxi", line 194, in cupy.core.core.compile_with_cache
  File "/shared-storage/username/miniconda/lib/python3.6/site-packages/cupy/cu
da/compiler.py", line 287, in compile_with_cache
    extra_source, backend)
  File "/shared-storage/username/miniconda/lib/python3.6/site-packages/cupy/cu
da/compiler.py", line 318, in _compile_with_cache_cuda
    os.makedirs(cache_dir)
  File "/shared-storage/username/miniconda/lib/python3.6/os.py", line 210, in 
makedirs
    makedirs(head, mode, exist_ok)
  File "/shared-storage/username/miniconda/lib/python3.6/os.py", line 210, in 
makedirs
    makedirs(head, mode, exist_ok)
  File "/shared-storage/username/miniconda/lib/python3.6/os.py", line 220, in 
makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/username'

I assume that it is a cache dir which cannot be created in /home/username Is there any chance I can change this cache dir to be create in /shared-storage/username ?


Solution

  • Set the env var CUPY_CACHE_DIR, see https://docs-cupy.chainer.org/en/stable/reference/environment.html.