Search code examples
pythonpandasnan

Pandas avoid element-wise NaN check for lists

I have a dataframe with columns containing both single NaNs, as well as lists which may contain NaN elements. Example:

df = pd.DataFrame({'A': [[1, 2, 3], np.nan, [7, np.nan, np.nan], [4, 5, 6]]})

I want to check whether the cell values are single NaNs. The expected result of [False, True, False, False] is printed by the following command:

df.isna()

Now I want to check whether a particular cell, say the third element (index 2) is NaN. Run the following code:

elem = df.loc[2,'A']
pd.isna(elem)

The output is array([ False, True, True]) rather than False. I understand that pd.isna() and np.isnan() go element-wise over lists or arrays (and that this is a sensible default), but is there a simple command to compare the list as a whole against a single NaN instead?

Solution

  • One simple option, especially if you plan to test multiple cells, would be to precompute a mask:

    mask = df.isna()

mask.loc[2, 'A']
# False

    If you really need to check a single value, ensure this is a float:

    x = df.loc[2, 'A']

isinstance(x, float) and pd.isna(x)
# False

    Or, force pd.isna to treat the input as a vector and aggregate with all:

    pd.isna([df.loc[2, 'A']]).all()