Serialize pandas dataframe consists NaN fields before sending as a response

Question

I have a dataframe that has NaN fields in it. I want to send this dataframe as a response. Because it has Nan fields I get this error,

ValueError: Out of range float values are not JSON compliant

I don't want to drop the fields or fill them with a character or etc. and the default response structure is ideal for my application.

Here is my views.py

...
forecast = forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']]
    forecast['actual_value'] = df['y']    # <- Nan fields are added here
    forecast.rename(
        columns={
            'ds': 'date',
            'yhat': 'predictions',
            'yhat_lower': 'lower_bound',
            'yhat_upper': 'higher_bound'
        }, inplace=True
    )

    

    context = {
        'detail': forecast
    }

    return Response(context)

Dataframe,

                         date  predictions  lower_bound  higher_bound  actual_value
0  2022-07-23 06:31:41.362011     3.832143    -3.256209     10.358063           1.0
1  2022-07-23 06:31:50.437211     4.169004    -2.903518     10.566005           7.0
2  2022-07-28 14:20:05.000000    12.085815     5.267806     18.270929          20.0
...
16 2022-08-09 15:07:23.000000   105.655997    99.017424    112.419991           NaN
17 2022-08-10 15:07:23.000000   115.347283   108.526287    122.152684           NaN

Hoping to find a way to send dataframe as a response.

Answer 1

You could try to use the fillna and replace methods to get rid of those NaN values.
Adding something like this should work as None values are JSON compliant:

forecast = forecast.fillna(np.nan).replace([np.nan], [None])

Using replace alone can be enough, but using fillna prevent errors if you also have NaT values for example.