drop a dictionary with nan value

Question

I have the following dictionary:

my_dict = {'fields': ['id': 1.0,
  'name': 'aaa',
  'type': 'string'},
 {'id': 3.0,
  'name': 'eee',
  'type': 'string'},

 {'id': nan,
  'name': 'bbb',
  'type': 'string'},

 {'id': 4.0,
  'name': 'ccc',
  'type': 'string'},

 {'id': nan,
  'name': 'ddd',
  'type': 'string'}],

'type': 'struct'
}

From this dictionary, I would like to drop the dictionary with the id value nan value and would like to get the following.

my_updated_dict = {'fields': ['id': 1.0,
  'name': 'aaa',
  'type': 'string'},

 {'id': 3.0,
  'name': 'eee',
  'type': 'string'},

 {'id': 4.0,
  'name': 'ccc',
  'type': 'string'}],

'type': 'struct'
}

I was trying changing to data frame and dropping the id value with the nan value and changing to dictionary back but couldn't get the intended result.

 my_updated_dict = pd.DataFrame(my_dict ).dropna().to_dict('list')

Answer 1

I do not know why would you need pandas for that if u can simply do:

my_dict["fields"] = [i for i in my_dict["fields"] if not np.isnan(i["id"])]

** UPDATE **

if you really do need for some reason to use pandas, you may try this constructiion:

my_dict["fields"] = pd.Series(my_dict["fields"]).apply(pd.Series).dropna().to_dict(orient="records")

though I do not see any advantages over simple list comprehension, except may be on big volume of information.