Making a faster wavelet transform/Appending data faster

Question

I am taking 1-D wavelet transform of data. How can I make this faster? I have 1.4 million samples and 32 features.

def apply_wavelet_transform(data):
    ca,cd=pywt.dwt(data[0,:],'haar')
    for i in range(1,data.shape[0]):
        ca_i,__=pywt.dwt(data[i,:],'haar')
        ca=np.vstack((ca,ca_i))
    return ca

Consider I don't care about memory usage as much as speed of execution.

Answer 1

This is a common mistake. You don't want to append rows to an array one at a time, because each iteration requires copying the entire array. Complexity: O(N**2). Much better to keep the intermediate results in a list and form the array at the end. This is better because lists do not require their elements to be contiguous in memory, so no copying is required.

def apply_wavelet_transform(data):
    results_list = []
    for row in data:
        ca, cd = pywt.dwt(row, 'haar')
        results_list.append(ca)
    result = np.array(results_list)
    return result