Get Numpy genfromtxt converter to use def function

Question

I have searched through the forums and cannot seem to work out the following problem. I am very new to python with a little programming experience so my issue might be trivial.

The want to convert a date-time string to date-time format using date.strptime classmethod.

The problem is the string format within the column is not consistent (majority are %Y-%m-%d $H:$M:$S.%f) ; when the time falls exactly on the second, the millisecond decimals are omitted (format should be instead %Y-%m-%d $H:$M:$S). When the strptime encounters an unrecognized format, it will simply place a None value in the array element.

Is there any way to create an exception in the lambda function (i.e., ValueError exception), if not, how do I pass the string value into a "normal" def timeConv(x) function from the genfromtxt converter option?

Maybe there is a better way of approaching this problem...?

My current code which results in a None value when the format is %Y-%m-%d $H:$M:$S:

timeConv = lambda x: datetime.strptime(x, '\"%Y-%m-%d $H:$M:$S.%f\"')

Time = np.genfromtxt(file, dtype='object', delimiter=',', skip_header=4, usecols=(0), converters = {0: timeConv})

Answer 1

You could use a try..except to first try one format, and if it does not work, catch the exception and try the other format:

import datetime as DT
import numpy as np

def timeConv(x):
    try:
        return DT.datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')
    except ValueError as err:
        return DT.datetime.strptime(x, '%Y-%m-%d %H:%M:%S')

time = np.genfromtxt(file, dtype='object', delimiter=',', skip_header=4,
                     usecols=(0), converters = {0: timeConv})

The function, timeConv is passed to genfromtxt the same way you passed the lambda.

---

The dateutil module has a date string parser which does not require you to specify the exact format of the date string. So using dateutil you could simply write

import dateutil.parser as dparser
import numpy as np

time = np.genfromtxt(file, dtype='object', delimiter=',', skip_header=4,
                     usecols=(0), converters = {0: dparser.parse})

Note that although dparser.parse is very easy to use, there are some ambiguous date strings like 2013-8-9 (August 8 or Sept 9?) which require more care. Be sure to read up on the dayfirst and yearfirst parameters so you can control the parser's behavior.