Search code examples
pythonpandasnumpymysql-python

Replacing Pandas or Numpy Nan with a None to use with MysqlDB

I am trying to write a Pandas dataframe (or can use a numpy array) to a mysql database using MysqlDB . MysqlDB doesn't seem understand 'nan' and my database throws out an error saying nan is not in the field list. I need to find a way to convert the 'nan' into a NoneType.

Any ideas?

Solution

  • For pandas > 1.3.0 see this answer.

    @bogatron has it right, you can use where, it's worth noting that you can do this natively in pandas:

    df1 = df.where(pd.notnull(df), None)

    Note: this changes the dtype of all columns to object.

    Example:

    In [1]: df = pd.DataFrame([1, np.nan])

In [2]: df
Out[2]: 
    0
0   1
1 NaN

In [3]: df1 = df.where(pd.notnull(df), None)

In [4]: df1
Out[4]: 
      0
0     1
1  None

    Note: what you cannot do recast the DataFrames dtype to allow all datatypes types, using astype, and then the DataFrame fillna method:

    df1 = df.astype(object).replace(np.nan, 'None')

    Unfortunately neither this, nor using replace, works with None see this (closed) issue.

    As an aside, it's worth noting that for most use cases you don't need to replace NaN with None, see this question about the difference between NaN and None in pandas.

    However, in this specific case it seems you do (at least at the time of this answer).