Search code examples
pythonscipyfftlarge-datadct

Python: SciPy DCT does not work on large matrices

I need to compute the DCT of a large matrix. My code seems to work fine for smaller matrices, but throws the following error for a matrix of size 50000 by 50000:

error: (n>0&&n<=size(x)) failed for the 1st keyword n: ddct2:n=50000

import numpy as np
from scipy import fftpack

# this works fine
y = fftpack.dct(np.random.normal(size = (1000,1000)))
# this throws an error
z = fftpack.dct(np.random.normal(size = (50000,50000)))

How to solve this? Many thanks.

Solution

  • It seems to work for me using scipy.fft (not fftpack):

    import numpy as np
import scipy.fft as fft

x = np.random.normal(size=(50000,50000))
y = fft.dct(x)

    Note, though, that a square ndarray of size 50000 will need upwards of 20 GB. You may run into issues processing that much data in one shot.

    Version notes: Python 3.9.2, NumPy 1.19.3, SciPy 1.6.1