Error in Writing Google Sheets using Pandas

Question

I have 2 data frames, df1 is user completion of courses, df2 is user information. I have combined them and reshaped using the following codes.

    # merge and use unstack to reshape the data
    df1=df1.merge(df2, on='User ID', how='left')
    df1 = df1.set_index(['User ID', 'Name', 'Course Name']).unstack().reset_index()
    # fix column names
    df1.columns = df1.columns.get_level_values(0).tolist()[:2] + df1.columns.get_level_values(1).tolist()[2:]

    print(df1)

User ID     Name    Lesson 1     Lesson 2    Lesson 3
0       1     John   Completed    Completed   Completed
1      10      Ray   Completed          NaN         NaN
2       2   Russel   Completed    Completed   Completed
3       3    Kathy   Completed  In Progress  Subscribed
4       4    Emily  Subscribed   Subscribed  Subscribed
5       5   George   Completed    Completed   Completed
6       6    Jercy   Completed  In Progress  Subscribed
7       7      Jay   Completed    Completed   Completed
8       8  Michael   Completed    Completed   Completed
9       9     Rose   Completed          NaN         NaN

But when I try to import the new data frame to my google sheets, the following error occurs:

    df_data = df1.to_numpy().tolist()
    headers = df1.columns.to_list()
    dataTowrite = [headers] + df_data
    datacombine = wb.worksheet("Completion")
    datacombine.update(None,dataTowrite)

APIError                                  Traceback (most recent call last)
<ipython-input-79-d242ae2a5864> in <module>
----> 1 datacombine.update(None,dataTowrite)

3 frames
/usr/local/lib/python3.7/dist-packages/gspread/utils.py in wrapper(*args, **kwargs)
    700                 kwargs.setdefault(k, v)
    701 
--> 702             return f(*args, **kwargs)
    703 
    704         return wrapper

/usr/local/lib/python3.7/dist-packages/gspread/worksheet.py in update(self, range_name, values, **kwargs)
   1042             params=params,
   1043             body=filter_dict_values(
-> 1044                 {"values": values, "majorDimension": kwargs["major_dimension"]}
   1045             ),
   1046         )

/usr/local/lib/python3.7/dist-packages/gspread/spreadsheet.py in values_update(self, range, params, body)
    224         """
    225         url = SPREADSHEET_VALUES_URL % (self.id, quote(range))
--> 226         r = self.client.request("put", url, params=params, json=body)
    227         return r.json()
    228 

/usr/local/lib/python3.7/dist-packages/gspread/client.py in request(self, method, endpoint, params, data, json, files, headers)
     90             return response
     91         else:
---> 92             raise APIError(response)
     93 
     94     def list_spreadsheet_files(self, title=None, folder_id=None):

APIError: {'code': 400, 'message': 'Invalid JSON payload received. Unexpected token.\n"Ray", "Completed", NaN, NaN], ["2", "Ru\n                    ^', 'status': 'INVALID_ARGUMENT'}

Answer 1

Since the error occured at the first line carrying NaN values, try to fill in those with empty strings.

Replace this:

df_data = df1.to_numpy().tolist()

By this :

df_data = df1.fillna("").to_numpy().tolist()