Modifying sparce matrix using fancy indexing

I am trying to use fancy indexing to modifying a large sparce matrix. Suppose you have the following code:

import numpy as np
import scipy.sparse as sp

a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
b = sp.lil_matrix(a)
c = sp.lil_matrix((3,4))
c[[1,2], 0] = b[[1,2], 0]

However, this code gives the following error:

ValueError: shape mismatch in assignment

I don't understand why this doesn't work. Both matrices have the same shape and this usually works if both matrices are numpy arrays. I would appreciate any help.

Solution

Yeah this is a bug with the sparse __setitem__. I've run into it before (but I just worked around it). Now I actually looked into it; first, you can fix this pretty easily:

import numpy as np
import scipy.sparse as sp

a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
b = sp.lil_matrix(a)
c = sp.lil_matrix((3,4))
c[[1,2], 0] = b[[1,2], 0]

This raises the ValueError you saw. This doesn't and works as expected:

c[[1,2], 0] = b[[1,2], [0]]

>>> c.A
array([[0., 0., 0., 0.],
       [5., 0., 0., 0.],
       [9., 0., 0., 0.]])

Lets just walk through the offending __setitem__ (I'm going to omit a lot of code that doesn't get called):

row, col = self._validate_indices(key)

This is fine - row = [1, 2] and col = 0

col = np.atleast_1d(col)
i, j = _broadcast_arrays(row, col)

So far so good - i = [1, 2] and j = [0, 0]

if i.ndim == 1:
# Inner indexing, so treat them like row vectors.
    i = i[None]
    j = j[None]

broadcast_row = x.shape[0] == 1 and i.shape[0] != 1
broadcast_col = x.shape[1] == 1 and i.shape[1] != 1

Here's our problem - i and j both got turned into row vectors with shape (1, 2). x here is what you're trying to assign (b[[1,2], 0]), which is of shape (2, 1); the next step raises a ValueError cause x and the indices don't align.

>>> c[[1,2], 0] = b[[1,2], 0].A

ValueError: cannot reshape array of size 4 into shape (2,)

Here's the same problem but __setitem__ broadcasts x into a (2,2) array, which then fails again because it's larger than the array you're assigning it to.

The workaround (b[[1,2], [0]]) has a shape of (1, 2) which is not correct, but that error ends up cancelling out the error in indexing c.

I'm not sure exactly what the logic is behind this indexing code so I'm not sure how to fix this without introducing other subtle bugs.