Search code examples
python-3.xpandasdataframepython-datetime

pandas TypeError: invalid type comparison when comparing datetime64 when using nunique

I tried to use nunique on datetime64, but got error,

TypeError: invalid type comparison

on the the code,

np.where(df.groupby('group_id')['payment_date'].transform('nunique') == 1, 0, 10)

I tried to check if the datetime64 values are all the same. The df looks like,

group_id    date
   1        2017-10-01
   1        2017-10-02
   1        2017-10-03
   1        2017-10-03
   2        2017-11-01
   2        2017-10-15
   2        2017-11-05

I am wondering how to fix the error.

Solution

  • I think this is an issue with the fact that the resultant output of nunique is coerced to datetime to preserve the dtype. Shouldn't happen if you ask me.

    The workaround is to do an nunique comparison on the string data:

    df.date.astype(str).groupby(df.group_id).transform('nunique')

    The rest follows from here as usual.