How to solve memory usage increase and slower in for loop

Question

Hi I'm doing some image processing. I have some problems when code running maybe... at least 1 hour code running well. But when the times on and on my code speed is getting slower and memory usage increase. I try to find some information about these problems. people use list comprehension or map func. Are these the only solutions?

x_array = np.array([])
y_array = np.array([])
present_x_array = np.array([])
present_y_array = np.array([])
cnt = 0

for x in range(curve.shape[1]):
    if np.max(curve[:, x]) > 200 :
        for y in range(curve.shape[0]):
            if curve[y,x] > 200 :
                present_x_array = np.append(present_x_array, x)
                present_y_array = np.append(present_y_array, y)
            else:
                continue
        if cnt == 0:
            x_array = present_x_array
            y_array = present_y_array
            cnt = cnt + 1
        else :
            if abs(np.max(y_array) - np.max(present_y_array)) <= 10 :
                x_array =np.append(x_array, present_x_array)
                y_array =np.append(y_array, present_y_array)
            else :
                continue
        present_x_array = np.array([])
        present_y_array = np.array([])

    else:
        continue

I try to make comprehension but it stucks handle 'cnt == 0' and 'cnt = cnt + 1'

Answer 1

x_array =np.append(x_array, present_x_array) - np.append creates a new copy whenever you append. Therefore, as the array size increase, the copy operation gets expensive. I would advise replacing np.array with list to avoid unnecessary copy operations. The list append is fast.

Reference: https://github.com/numpy/numpy/issues/17090