While trying to do ndimage.convolve on big numpy.memmap, exception occurs:
Exception has occurred: _ArrayMemoryError Unable to allocate 56.0 GiB for an array with shape (3710, 1056, 3838) and data type float32
Seems that convolve creates a regular numpy array which won't fit into memory.
Could you tell me please if there is a workaround?
Thank you for any input.
Instead of running your own implementation, another approach that might work would be to use dask's wrapped ndfilters with a dask array created from the memmap. That way, you can delegate the chunking/out-of-memory-calculation parts to Dask.
I haven't actually done this myself, but I see no reason why it wouldn't work!