How to iterate over column names with PyTable?

Question

I have a large matrix (15000 rows x 2500 columns) stored using PyTables and getting see how to iterate over the columns of a row. In the documentation I only see how to access each row by name manually.

I have columns like:

• ID
• X20160730_Day10_123a_2
• X20160730_Day10_123b_1
• X20160730_Day10_123b_2

The ID column value is a string like '10692.RFX7' but all other cell values are floats. This selection works and I can iterate the rows of results but I cannot see how to iterate over the columns and check their values:

from tables import *
import numpy

def main():
    h5file = open_file('carlo_seth.h5', mode='r', title='Three-file test')
    table = h5file.root.expression.readout

    condition = '(ID == b"10692.RFX7")'
    for row in table.where(condition):
        print(row['ID'].decode())

        for col in row.fetch_all_fields():
            print("{0}\t{1}".format(col, row[col]))

    h5file.close()

if __name__ == '__main__':
    main()

If I just iterate with "for col in row" nothing happens. As the code is above, I get a stack:

10692.RFX7
Traceback (most recent call last):
  File "tables/tableextension.pyx", line 1497, in tables.tableextension.Row.__getitem__ (tables/tableextension.c:17226)
KeyError: b'10692.RFX7'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tables/tableextension.pyx", line 126, in tables.tableextension.get_nested_field_cache (tables/tableextension.c:2532)
KeyError: b'10692.RFX7'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./read_carlo_pytable.py", line 31, in <module>
    main()
  File "./read_carlo_pytable.py", line 25, in main
    print("{0}\t{1}".format(col, row[col]))
  File "tables/tableextension.pyx", line 1501, in tables.tableextension.Row.__getitem__ (tables/tableextension.c:17286)
  File "tables/tableextension.pyx", line 133, in tables.tableextension.get_nested_field_cache (tables/tableextension.c:2651)
  File "tables/utilsextension.pyx", line 927, in tables.utilsextension.get_nested_field (tables/utilsextension.c:8707)
AttributeError: 'numpy.bytes_' object has no attribute 'encode'
Closing remaining open files:carlo_seth.h5...done

Answer 1

You can access a column value by name in each row:

for row in table:
    print(row["10692.RFX7"])

Iterate over all columns:

names = table.coldescrs.keys()
for row in table:
    for name in names:
        print(name, row[name])