Search code examples
pythonchainercupy

UnicodeDecodeError When I use cuda to train dataset


I used chainer to train some images but there is an error.

I don't know whether its UnicodeDecodeError or the error of installation of cupy.

P:\dcgans\chainer-DCGAN\chainer-DCGAN>python DCGAN.py

Traceback (most recent call last):
  File "DCGAN.py", line 279, in <module>
    train_dcgan_labeled(gen, dis)
  File "DCGAN.py", line 171, in train_dcgan_labeled
    zvis = (xp.random.uniform(-1, 1, (100, nz), dtype=np.float32))
  File "P:\Python35\lib\site-packages\cupy\random\distributions.py", line 132, in uniform
    return rs.uniform(low, high, size=size, dtype=dtype)
  File "P:\Python35\lib\site-packages\cupy\random\generator.py", line 235, in uniform
    rand = self.random_sample(size=size, dtype=dtype)
  File "P:\Python35\lib\site-packages\cupy\random\generator.py", line 153, in random_sample
    RandomState._1m_kernel(out)
  File "cupy/core/elementwise.pxi", line 552, in cupy.core.core.ElementwiseKernel.__call__ (cupy\core\core.cpp:43810)
  File "cupy/util.pyx", line 39, in cupy.util.memoize.decorator.ret (cupy\util.cpp:1480)
  File "cupy/core/elementwise.pxi", line 409, in cupy.core.core._get_elementwise_kernel (cupy\core\core.cpp:42156)
  File "cupy/core/elementwise.pxi", line 12, in cupy.core.core._get_simple_elementwise_kernel (cupy\core\core.cpp:34787)
  File "cupy/core/elementwise.pxi", line 32, in cupy.core.core._get_simple_elementwise_kernel (cupy\core\core.cpp:34609)
  File "cupy/core/carray.pxi", line 87, in cupy.core.core.compile_with_cache (cupy\core\core.cpp:34264)
  File "P:\Python35\lib\site-packages\cupy\cuda\compiler.py", line 133, in compile_with_cache
    base = _empty_file_preprocess_cache[env] = preprocess('', options)
  File "P:\Python35\lib\site-packages\cupy\cuda\compiler.py", line 99, in preprocess
    pp_src = pp_src.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 27-28: invalid continuation byte

Solution

  • It seems nvcc generated non-UTF8 output and CuPy failed to decode it. This is a bug of CuPy (I posted an issue: #378).

    A possible solution for the time being is to replace 'utf-8' in cupy/cuda/compiler.py at the line pp_src = pp_src.decode('utf-8') with something that match your environment. For example, in Japanese environment, 'cp932' should work, and 'cp936' should perhaps work for simplified Chinese.

    You could also try locale.getdefaultlocale()[1] as a universal solution (be sure to import locale).

    Update: The fix has been merged. It should be fixed in upcoming CuPy v1.0.3.