Search code examples
pythonpandasdataframenan

How to filter a Boolean mask containing NaNs


I have this pandas DataFrame bellow:

    Id                          Guild                                     Test
0   5c5dc770f920209b94c3def3    72f92390/7f2e/4b41/b53b/393470619eca      True
1   5c5dc7707d62f8b356457863    596f57d7/c8a9/4b14/aec1/18ef2b9fa940      None
2   5c5dc770974d1a6d38cffa3a    6a7ad94c/0511/4ef9/8b60/e05158cad03c     False
3   5c5dc7709809c3452ae07d22    843d9c5f/1f53/4752/a905/0b1de73efab2      None
4   5c5dc7706c606a2118c4350b    9d63dcc5/1063/49b3/9a90/a854e7eb7398      None

When i tried to apply numpy.where:

pdf['Id'] = np.where(bool(pdf['Test']), pdf['Id'], None)

Also tried using numpy.equal:

pdf['Id'] = np.where(np.equal(pdf['Test'], None), None, pdf['Id'])

Throws me the error:

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

My goal: Apply None to Id columns where Teste is not a valid boolean.

I've check these similar questions: link One Link Two

Thanks in advance.


Solution

  • The complication here is that your "boolean" column also has None values.

    You can instead compare the "Test" column to True.

    pdf['Id'] = np.where(pdf['Test'] == True, pdf['Id'], None)
    pdf    
                             Id                                 Guild   Test
    0  5c5dc770f920209b94c3def3  72f92390/7f2e/4b41/b53b/393470619eca   True
    1                      None  596f57d7/c8a9/4b14/aec1/18ef2b9fa940   None
    2                      None  6a7ad94c/0511/4ef9/8b60/e05158cad03c  False
    3                      None  843d9c5f/1f53/4752/a905/0b1de73efab2   None
    4                      None  9d63dcc5/1063/49b3/9a90/a854e7eb7398   None
    

    Or, assign using loc.

    pdf.loc[pdf['Test'] != True, 'Id'] = None
    pdf    
                             Id                                 Guild   Test
    0  5c5dc770f920209b94c3def3  72f92390/7f2e/4b41/b53b/393470619eca   True
    1                      None  596f57d7/c8a9/4b14/aec1/18ef2b9fa940   None
    2                      None  6a7ad94c/0511/4ef9/8b60/e05158cad03c  False
    3                      None  843d9c5f/1f53/4752/a905/0b1de73efab2   None
    4                      None  9d63dcc5/1063/49b3/9a90/a854e7eb7398   None