Search code examples
pythonnumpynumpy-memmap

I can't remove file, created by memmap

I can't remove file created numpy.memmap funtion

class MyClass
 def __init__(self):    
  self.fp = np.memmap(filename, dtype='float32', mode='w+', shape=flushed_chunk_shape)
    ...
 def __del__(self):   
  del self.fp
  os.remove(filename)

When I run del myclass (instanse of MyClass) I've got error WindowsError: [Error 32] The process cannot access the file. If I use memmap and deleting the file in no objected oriented way everything ok. So I consider trouble in destructor. But why it happens?

Solution

  • Numpy memmapped files are not unmapped until garbage collected, and del self.fp does not ensure garbage collection. A file which is not unmapped yet cannot be deleted.

    The numpy.memmap docs say "Deletion flushes memory changes to disk". It does not say it unmaps, because it doesn't unmap.

    Solution 1. Force garbage collection 

    import gc
...
del self.fp
gc.collect()
os.remove(filename)

    You need to make sure there are no other references to fp anywhere, or it may not be garbage collected even in this case.

    Solution 2. Force unmap

    self.fp._mmap.close()
del self.fp
os.remove(filename)

    Note that in the second case, if there is an access to fp after it's closed, it will probably crash the python interpreter (not throw exception, just crash)