Append new columns to HDFStore with pandas

Question

I'm using Pandas, and making a HDFStore object. I calculate 500 columns of data, and write it to a table format HDFStore object. Then I close the file, delete the data from memory, do the next 500 columns (labelled by an increasing integer), open up the store, and try to append the new columns. However, it doesn't like this. It gives me an error

invalid combinate of [non_index_axes] on appending data [[(1, [500, 501, 502, ...])]] vs current table [[(1, [0, 1, 2, ...])]]

I'm assuming it only allows appending of more rows not columns. So how do I add more columns?

Answer 1

HDF5 files have a fixed structure, and you cannot easily add a column , but the workaround is to concatenate different DFs and the re-write them into the HDF5 file.

hdf5_files = ['data1.h5', 'data2.h5', 'data3.h5']

df_list = []
for file in hdf5_files:
    df = pd.read_hdf(file)
    df_list.append(df)

result = pd.concat(df_list)

# You can now use the result DataFrame to access all of the data from the HDF5 files

Does this solve your problem ?

Remind HDF5 is not designed for efficient append operations, you should consider database system if you need to frequently add new columns to your data , imho.