How can I apply dynamic date formats to multiple types in Elasticsearch?

Question

I'm trying to get Elasticsearch to recognise strings in the format yyyy-MM-dd HH:mm:ss as date fields. I've created a dynamic date format and applied that to the default mapping. It works nicely when I index documents of my first type - any new fields where the data is in this format get initialised as date fields.

The problem comes when I try to create documents of a new type, but with date format fields with the same name as in my first type. These fail with a malformed date error.

Here's an example set of Kibana commands to demonstrate:

DELETE /datetest

PUT /datetest

PUT /datetest/_mapping/_default_
{
  "dynamic_date_formats" : ["yyyy-MM-dd HH:mm:ss"]
}

PUT /datetest/doc/1
{
  "date" : "2015-01-01 12:00:00"
}

# This one works fine
PUT /datetest/otherdoc/1
{
  "otherdate" : "2015-01-01 12:00:00"
}

# This one does not
PUT /datetest/otherdoc/2
{
  "date" : "2015-01-01 12:00:00"
}

The last command gives this error: "Invalid format: \"2015-01-01 12:00:00\" is malformed at \" 12:00:00\""

I know that fields with the same name in different types must have the same data type, but in this case, I want them to have the same data type - date. I could manually create the mappings for each new type, but I want it to automatically support new types added to my source data. It seems to be what the dynamic date format is supposed to do. Am I doing something wrong here?

Answer 1

I would create a custom dynamic template. Something like this:

PUT /datetest/_mapping/_default_
{
    "date_detection" : true,
    "dynamic_templates" : [
        {
            "dates" : {
               "match" : ".*date.*",
                "mapping" : {
                    "type" : "date",
                    "format" : 'yyyy-MM-dd HH:mm:ss'
                }
            }
        }
    ]
}

Just tried it. It seems to work. I hope this helps :)