Search code examples
pythongoogle-app-enginebulkloader

precision of DateTimeProperty in appengine bulkload

I'm trying to download data from my datastore using the Bulkloader, but the precision of the DateTimeProperty seems off, unless I'm doing something the wrong way.

This is the property of the data I'm trying to download:

class LoggingPrimitive(polymodel.PolyModel):
    [...]
    start = ndb.DateTimeProperty(auto_now_add=True, required=True)

This is my bulkloader.yaml file

- kind: LoggingPrimitive
  connector: csv
  connector_options:
    export_options:
      delimiter: "|"
  property_map:
    [...]
    - property: start
      external_name: start
      # Type: Date/Time Stats: 5123 properties of this type in this kind.
      import_transform: transform.import_date_time('%Y-%m-%dT%H:%M:%S')
      export_transform: transform.export_date_time('%Y-%m-%d %H:%M:%S.%f')

The data in my downloaded CSV look like:

    2013-05-25 22:06:17.

So, the downloaded CSV data does not contain the milliseconds.

When I use the transform.export_date_time function in the interactive console, I do get the milliseconds.

fn=transform.export_date_time('%Y-%m-%d %H:%M:%S.%f')
print fn(l.start)

So the milliseconds information is in the datastore, but seems to get lost somewhere in the bulkload process.

Any clues?

Solution

  • Running the bulkloader with Python 2.7 resolved this issue.

    Apparently the machine I was running the bulkloader from was using Python 2.5 , which does not support the %f macro, see How can I parse a time string containing milliseconds in it with python?