Reading numpy array from file and parsing very slow

Question

I have a binary file and I am parsing it to a numpy array in Python like the following:

bytestream= np.fromfile(path, dtype=np.int16)

 for a in range(sizeA):
        for x in range(0, sizeX):
            for y in range(0, sizeY):
                for z in range(0, sizeZ):
                    parsed[a, x, y, z] = bytestream[z + (sizeZ * x) + (sizeZ * sizeX * y) + (sizeZ * sizeX * sizeY * a)]

However, this is very very slow. Can anyone tell me why and how to speed it up?

Answer 1

You seem to have made a mistake in your code, I believe x and y should be reversed in (sizeZ * x) + (sizeZ * sizeX * y) assuming row major ordering. In any case, check the code below, which verifies that reshape is what you want. The reason it's slow is because of the nested for loops.

In python, for loop is a very complicated construct with very significant overhead. Therefore in most cases, you should avoid for loops and use library provided functions(which also have for loops but done in c/c++). You will find that "removing the for loop" is a common question in numpy as most people will first attempt some algorithm they know in most straight forward way(convolution, max pooling for example). And realized that it's very slow and look for clever alternatives based on numpy api, where majority of computation shifted to c++ side instead of happening in python.

import numpy as np

# gen some data 
arr= (np.random.random((4,4,4,4))*10).astype(np.int16)
arr.tofile('test.bin')

# original code
bytestream=np.fromfile('test.bin',dtype=np.int16)
parsed=np.zeros(arr.shape,dtype=np.int16)
sizeA,sizeX,sizeY,sizeZ=arr.shape
for a in range(sizeA):
    for x in range(0, sizeX):
        for y in range(0, sizeY):
            for z in range(0, sizeZ):
                parsed[a, x, y, z] = bytestream[z + (sizeZ * y) + (sizeZ * sizeX * x) + (sizeZ * sizeX * sizeY * a)]

print(np.allclose(arr,parsed))
print(np.allclose(arr,bytestream.reshape((sizeA,sizeX,sizeY,sizeZ))))