I'm solving a problem that involves a sparse matrix. It has the three main diagonals, and a bunch of other subdiagonals. The full size of the matrix is (2048000, 2048000), but as it is quite sparse, it has only 525312000 stored elements, corresponding to about 4 GB of memory for double precision. When I create this matrix, Activity Monitor and top on my Mac both report a memory use of about 4 GB, as expected.
Next, I create an incomplete LU factorisation, to use as a preconditioner when solving the matrix system with BiCGStab. I use the following code:
from scipy.sparse.linalg import spilu
ILU = spilu(csc_matrix(L+Lr))
here, Lr is the matrix I mentioned above, L is another sparse diagonal matrix that is purely tridiagonal, and thus much smaller.
The variable ILU is of type SuperLU, and according to ILU.nnz it contains only 20384063 stored elements, which means it should take about 150 MB of memory, yet Activity Monitor and top both claim that I am now using about 8 GB of memory, where previously I was using about 4 GB. So what happened to all of that memory?
Not a particularly satisfying answer, but I posted an issue at the SciPy repo, and from the discussion it seems this is an issue on Mac only, and that there isn't much to be done about it.