How to re-format a date-time string in nxlog or how to set up date-time format for python logging file

Whilst it's a two question post, the answer to either one will fix my single problem, so I only need a response to one of the two questions.

I'm using an app called dynamic-dynamodb do scale our dynamo tables.

The next thing is to get the logs from that into logstash via nxlog, I've done that as well.

The problem is that the date-time format is slightly wrong, it's

2015-02-24 14:55:39,777

instead of (note the T and the .):

2015-02-24T14:55:39.777

Frankly speaking, the documentation on how to configure the logging output is kind of.. well it's rubbish.

I was able to work out how to format it as json but I can't work out how to change the date-time format as it stands.

Here's my format (the format string is stored as a heredoc in a hash and then written to the conf as a template in chef):

{
    "EventReceivedTime": "%(asctime)s",
    "name":  "%(name)s",
    "level": "%(levelname)s",
    "message": "%(message)s",
    "type":"dynamic-dynamodb",
    "hostname": "#{node['hostname']}",
    "enviroment" : "#{node.chef_environment}",
    "node_name" : "#{app_name}",
    "ipaddress" : "#{node['ipaddress']}"
}

So, since I can't work out how to change the format I thought I'd re-format the date-time string via nxlog (which reads the log in as json) before I push to logstash but I've looked and well, I can't seem to find how to do that either.

So, any help to answer either of the problems would be fantastic.

1) How to re-format the time directly in the logger format

2)How to re-format the date-time in nxlog before I push to logstash

Solution

I was able to work out how to change the date-time format that the logger outputs as. I added to the logfile config

datefmt=%Y-%m-%dT%H:%M:%S

and changed the formatter to:

"
{
    'EventReceivedTime': '%(asctime)s.%(msecs)d',
    'name':  '%(name)s',
    'level': '%(levelname)s',
    'message': '%(message)s',
    'type':'dynamic-dynamodb',
    'hostname': '#{node['hostname']}',
    'enviroment' : '#{node.chef_environment}',
    'node_name' : '#{app_name}',
    'ipaddress' : '#{node['ipaddress']}'
}
".gsub('\'','"')#<-- this is so I can juse use ' rather than having to \"

The reason for the change from heredoc to multiline string was because I couldn't add a second element to the hash properly, this was the only way.

The reason for gsub was because It was easier to gsub every ' for " then having to escape every " with \".

I used information from here to realise that I could just add datefmt to the log file and it would change the datetime template. Specifically it was the presence of datefmt within both the yaml format and the json format that made me realise I could just do the same for the logger config.