List of dictionarys to dataframe

Question

im trying to make a dataframe out of a list of dictionaries. I am quite new at this whole programming thing, and google just makes me more confused. That is why i am turning to you guys hoping for some assistance. The first two list values (YV01', '3nP3RFgGnBrOfILK4DF2Tp) i would like to have under columns called: Name and GlobalId. I would lie to drop Pset_wallcommon, AC_Pset_RenovationAndPhasing, and BaseQuantities. And use the rest of the keys(if that what they are called) as column names.

It would be great if someone could give me the right push :)

For the record: Im am parsing an Ifc file with the IfcOpenshell package

The data:
['YV01', '3nP3RFgGnBrOfILK4DF2Tp', {'Pset_WallCommon': {'Combustible': False, 'Compartmentation': False, 'ExtendToStructure': False, 'SurfaceSpreadOfFlame': '', 'ThermalTransmittance': 0.0, 'Reference': '', 'AcousticRating': '', 'FireRating': '', 'LoadBearing': False, 'IsExternal': False}, 'AC_Pset_RenovationAndPhasing': {'Renovation Status': 'New'}, 'BaseQuantities': {'Length': 13786.7314346, 'Height': 2700.0, 'Width': 276.0, 'GrossFootprintArea': 3.88131387595, 'NetFootprintArea': 3.88131387595, 'GrossSideArea': 37.9693748734, 'NetSideArea': 37.9693748734, 'GrossVolume': 10.4795474651, 'NetVolume': 10.4795474651}}, 'YV01', '1M4JyBJhXD5xt8fBFUcjUU', {'Pset_WallCommon': {'Combustible': False, 'Compartmentation': False, 'ExtendToStructure': False, 'SurfaceSpreadOfFlame': '', 'ThermalTransmittance': 0.0, 'Reference': '', 'AcousticRating': '', 'FireRating': '', 'LoadBearing': False, 'IsExternal': False}, 'AC_Pset_RenovationAndPhasing': {'Renovation Status': 'New'}, 'BaseQuantities': {'Length': 6166.67382573, 'Height': 2700.0, 'Width': 276.0, 'GrossFootprintArea': 1.6258259759, 'NetFootprintArea': 1.6258259759, 'GrossSideArea': 15.9048193295, 'NetSideArea': 15.9048193295, 'GrossVolume': 4.38973013494, 'NetVolume': 4.38973013494}}

all_walls = ifc_file.by_type('IfcWall')
wallList = []

for wall in all_walls:
    propertySets = (ifcopenshell.util.element.get_psets(wall))
    wallList.append(wall.Name)
    wallList.append(wall.GlobalId)
    wallList.append(propertySets)
print(wallList)

wall_table = pd.DataFrame.from_records(wallList)
print(wall_table)

I have tried these basic pd.DataFrame.from_dict/records/arrays(data) but the output looks like this

enter image description here

UPDATE: Thank you so much for your help, i am learning alot from this! So i made a dictionary out of the wallList, and flattened the dict. like this:

 #list of walls
    for wall in all_walls:
        propertySets = (ifcopenshell.util.element.get_psets(wall))
        wallList.append(wall.Name)
        wallList.append(wall.GlobalId)
        wallList.append(propertySets)
    
    #dict from list
    wall_dict = {i: wallList[i] for i in range(0, len(wallList))}
    
    new_dict = {}
    #flattening dict
    for key, value in wall_dict.items():
        if isinstance(value, dict):
            for key in value.keys():
                for key2 in value[key].keys():
                    new_dict[key + '_' + key2] = value[key][key2]
        else:
            new_dict[key] = value
    
    
    wall_table = pd.DataFrame.from_dict(new_dict, orient='index')
    
    print(wall_table)

It seems to work pretty good, the only problem is that the dataframe contains all walls, but only propertyset data from the first in the list. I cant seem to understand how the dict flattening loop works. I would also like the index names (Pset_WallCommon_Combustible, and so on) to be the columns in my dataframe. Is that possible?

enter image description here

Answer 1

EDIT : Simply flattening a list as i did goes nowhere. Actually, i think you should drop this list thing altogether and try to load the Dataframe from a dictionnary. We'd need to see what does all_walls look like to help you for that, tho.

Have you tried directly loading the all_walls dictionary into a dataframe : df = pd.Dataframe.from_dict(all_walls) ?

I think if that doesnt work, flattening the dictionnaries in a fashion similar to the following should do the trick.

new_dict = {}
for key, value in all_walls.items():
    if isinstance(value, dict):
        for key in value.keys():
            for key2 in value[key].keys():
                new_dict[key + '_' + key2] = value[key][key2]
    else:
        new_dict[key] = value