NumPy genfromtxt for different column sizes

Question

I am attempting to extract values from a txt file using numpy.genfromtxt() method. MY txt file looks like this:

 '! dt         tot            nsave     readext\n',
 '  0.002      200            500       F\n',
 '!----------------------------------------------------------------------\n',
 '! xdomain       ydomain \n',
 '  7.5           7.5\n',
 '!----------------------------------------------------------------------\n',
 '! maxnewts  maxiters  atol\n',
 '  40        100       0.001\n',
 '!----------------------------------------------------------------------\n',
 '! p      \n',
 '  600  \n',

But using numpy.genfromtxt("file.txt", comments='!') gives me:

Line #4 (got 2 columns instead of 4)
Line #7 (got 3 columns instead of 4)
Line #10 (got 1 columns instead of 4)

How can I make numpy.genfromtxt flexible about the column sizes?

Answer 1

It seems the text file is not in the right format for analysis. I would suggest using csv to get what you need out of the file and then do your processing with numpy

import csv
clean_vars = []
with open('foo.txt', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter=' ', quotechar=",")
    for row in reader:
        # clean up your file here and append it to the list
        clean_vars.append([char for char in row if char])

# do your processing with numpy with your clean vars

In the case above, I am cleaning the vars inefficiently as an example. The docs for csv can be found at https://docs.python.org/2/library/csv.html