Python txt matrix from multiple files

Question

How can I convert line wise frequency distributions from multiple TXT files into a single matrix? Each of the files has exactly the same structure in that all words/terms/phrases are in the same order and contained in every file. Unique for each file is the filename, an issue date and the respective frequency of the words/terms/phrases given by a number after ":", see the following:

How my input files look like:

FilenameA Date:31.12.20XX
('financial' 'statement'):15
('corporate-taxes'):3
('assets'):8
('available-for-sale' 'property'):2
('auditors'):23

I have multiple files which have the exact same order of words/phrases and only differ in the frequency (number behind ":")

Now I want to create a single file containing a matrix, which keeps all words as top column and attaches the file characteristics (filename, date and frequencies) as row wise entries:

Desired Output:

Filename  Date  ('financial' 'statement') ('corporate-taxes') ... ('auditors)
A         2008             15                      3                  23
B         2010              9                      6                  11
C         2013              1                      8                   4
...
.
.

Really appreciate any help, would be great to have a loop which reads all files from a directory and outputs the above matrix.

Answer 1

The following code should help you:

import os

# Compute matrix
titles = ['Filename', 'Date']
matrix = [titles]
for directory, __, files in os.walk('files'): # replace with your directory
    for filename in files:
        with open(os.path.join(directory, filename)) as f:
            name, date = f.readline().strip().split()
            row = [name[8:], date.split('.')[-1]]
            for line in f:
                header, value = line.strip().split(':')
                if len(matrix) == 1:
                    titles.append(header)
                row.append(value)        
        matrix.append(row)

# Work out column widths
column_widths = [0]*len(titles)
for row in matrix:
    for column, data in enumerate(row):
        column_widths[column] = max(column_widths[column], len(data))
formats = ['{:%s%ss}' % ('^' if c>1 else '<', w) for c, w in enumerate(column_widths)]

# Print matrix
for row in matrix:
    for column, data in enumerate(row):
        print formats[column].format(data), 
    print

Sample output:

Filename Date ('financial' 'statement') ('corporate-taxes') ('assets') ('available-for-sale' 'property') ('auditors')
A        2012            15                      3              8                      2                      23     
B        2010             9                      6              8                      2                      11     
C        2010             1                      8              8                      2                      4