Search code examples
pythonpytorchoserror

OSError: [Errno 22] Invalid argument when using torch.load

I am trying to load my dataset and it was working before but all of the sudden this error started pooping up.

When I try to load it like this:

  train_set = Database_load(root = "C:\\Users\\Public\\PhysNet\\",train = "train.pth")

It gives me the following error:


  File "C:\Users\Public\Lucas\PhysNet\modt.py", line 185, in <module>
    train_set = Database_load(root = "C:\\Users\\Public\\Lucas\\PhysNet\\",train = "train.pth")

  File "C:\Users\Public\Lucas\PhysNet\database_load.py", line 21, in __init__
    self.data, self.y1, self.y2= torch.load(os.path.join(self.root, self.train))

  File "C:\ProgramData\Anaconda3\envs\project\lib\site-packages\torch\serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)

  File "C:\ProgramData\Anaconda3\envs\project\lib\site-packages\torch\serialization.py", line 580, in _load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)

OSError: [Errno 22] Invalid argument

I have tried doing the following as well when entering the path but it has not worked:

  train_set = Database_load(root = r"C:\\Users\\Public\\PhysNet\\",train = "train.pth")

  train_set = Database_load(root = r'C:/Users/Public/PhysNet/',train = 'train.pth')

Any suggestions on how to fix this issue?

Solution

  • This was a known issue (Issue#26998 and Issue#) caused by PR#20900. The problem happens because you're trying to load a file larger than 2GB, and it is specific to Windows in which "sizeof(long)=4 for both 32-bit and 64-system systems". This issue was fixed by PR#27069 and is only available in PyTorch 1.3+. Therefore, to fix this issue, please upgrade your PyTorch version.