numpy genfromtxt converters unknown number of columns

Question

I have several data numeric files in which the decimal separator is a comma. So I use a lambda function to do a conversion:

import numpy as np
def decimal_converter(num_cols):
    conv = dict((col, lambda valstr: \
    float(valstr.decode('utf-8').replace(',', '.'))) for col in range(nb_cols))
    return conv

data = np.genfromtxt("file.csv", converters = decimal_converter(3))

the data in the file is like this:

0; 0,28321815;  0,5819178
1; 0,56868281;  0,85621369
2; 0,24022026;  0,53490058
3; 0,63641921;  0,0293904
4; 0,65585546;  0,55913776

Here with my function decimal_converter I need to specify the number of columns my file contains. Normally I don't need to specify numpy.genfromtxt the number of columns in the file and it takes all it finds. I would like to keep this feature even when using converters option.

Answer 1

Since genfromtxt() accepts an iterator, you can pass the iterator applying your conversion function and then you can avoid the converters parameter:

import numpy as np

def conv(x):
    return x.replace(',', '.').encode()

data = np.genfromtxt((conv(x) for x in open("test.txt")), delimiter=';')