Search code examples
pythonnumpynumpy-memmap

numpy.memmap returns not enough memory while there are plenty available

During a typical call to numpy.memmap() on a 64bit windows machine, python raise the following error:

OSError: [WinError 8] Not enough memory resources are available to process this command

A different windows machine raise the same error with a different text:

OSError: [WinError 8] Not enough storage is available to process this command.

Here is the code abstract:

with open(infile, 'rb') as f:
  ......
  array = numpy.memmap(f, dtype='uint8', mode='r', offset=offset, shape=arraysize).tolist()

Python only used 50MB of the memory by this time. What would be the cause of running out of memory?

Solution

  • It turns out the issue here is that the offset + shape in the memmap call is greater than the total size of the file (i.e. I am trying to read beyond the size of the file).

    The error message about memory resource is a little misleading in this case.