In my dataframe I have a column named 'items'(list like string) like below
which is of type string. I want to convert this into a list . I have tried various methods like ast.literal_eval ,eval etc..
import ast
import json
import numpy as np
null= None
d['items']=d['items'].apply(lambda x:ast.literal_eval(str(x)) if(np.all(pd.notnull(x))) else x )
By using ast.literal_eval method I am getting this error -ValueError: malformed node or string: <_ast.Name object at 0x0000019AE0EA4B80>
By using eval() method I get this error - File "", line unknown SyntaxError: unexpected EOF while parsing
d['items']=d['items'].apply(lambda x:ast.literal_eval(str(x)) if(np.all(pd.notnull(x))) else x )
I have data with 139358 rows. In this data there are chances of having null, None or any other elements that could have caused this EOF error. I am assuming I had properly handled the nans using the if condition -('if(np.all(pd.notnull(x)))').
Please suggest any exception handling to do if that can help or let me know any solution that can convert these list-like-strings to lists.
Finally the datatype of records in items should be list but not strings (string to list conversion) Thanks in advance!
Answer: My data had Invalid JSON objects. So written an if statement to filter out the invalid JSONS
Make sure if the data is a perfect JSON or not