Replacing Square Brackets without value in .json array/pandas dataframe

Question

I was wondering if there is a way to remove/replace null/empty square brackets in json or pandas dataframe. I have tried to replace them after converting into string via .astype(str) and it is successful and/but it seems it converts all json values into string and I can not process further with the same structure. I would appreciate any solution/recommendation. thanks...

Answer 1

With the following toy dataframe:

import pandas as pd

df = pd.DataFrame({"col1": ["a", [1, 2, 3], [], "d"], "col2": ["e", [], "f", "g"]})

print(df)
# Output

Here is one way to do it:

df = df.applymap(lambda x: pd.NA if isinstance(x, list) and not x else x)

print(df)
# Output