Similar questions to this have been asked many times, but surprisingly few of the answers seem to address what I believe my problem to be.
I have csv files with one or more columns that contain a dictionary in each cell. After read_csv step, I have tried ast.literal_eval on these columns in order to convert the str format dicts into dict type objects. However, I keep getting the malformed node or string error.
Here is a typical example of the dicts in question: {1: 3681.45, 0: 3693.3333333333335}. And another, with a nan value: {1: 4959.95652173913, 0: nan}. Edit: It was only this nan value causing the error, in fact (see Rakesh's solution below).
Looking through previous answers, one reason for this error may be because most of the values of the dicts in these columns are floating point numbers, and apparently literal_eval can't handle floats or nans, even if they're contained within dictionaries (although, this is me inferring from a question about tuples).
I had to read a lot of questions and answers even to get this much information, so thought it could be useful to start a new topic with the keywords str, dict, but also nan in the title.
Any help much appreciated!
Use eval on json that has nan
Ex:
import ast
from numpy import nan
print(ast.literal_eval("{1: 3681.45, 0: 3693.3333333333335}"))
print(eval("{1: 4959.95652173913, 0: nan}"))
df = pd.DataFrame({"A": ["{1: 4959.95652173913, 0: nan}", "{1: 4959.95652173913, 0: nan}"]})
df['B'] = df["A"].apply(lambda x: eval(x))
print(df)
Output:
A B
0 {1: 4959.95652173913, 0: nan} {1: 4959.95652173913, 0: nan}
1 {1: 4959.95652173913, 0: nan} {1: 4959.95652173913, 0: nan}