Filebeat date field mapped as type keyword

Question

Filebeat is reading logs from a file, where logs are in the following format:

{"logTimestamp":"2019-11-29T16:39:43.027Z","@version":"1","message":"Hello world","logger_name":"se.lolotron.App","thread_name":"thread-1","level":"INFO","level_value":40000,"application":"my-app"}

So there is a field logTimestamp logged in ISO 8601 time format. The problem is that this field is mapped as a keyword In Elasticsearch filebeat index

"logTimestamp": {
    "type": "keyword",
    "ignore_above": 1024
},

On the other hand if I index a similar document in the same Elasticsearch instance but different index, e.g.

POST /new_index/_doc/
{
    "message": "hello world",
    "logTimestamp":"2019-11-29T16:39:43.027Z"
}

The mapping is

"logTimestamp": {
     "type": "date"
},

According to docs here and here by default Elastic should detect a date if formatted with strict_date_optional_time. And strict_date_optional_time is described as

A generic ISO datetime parser where the date is mandatory and the time is optional.

Which I presume is ISO 8601 and think I proved that with indexing a new doc to new_index in the example above.

Why is logTimestamp saved as keyword in the case of Filebeat? Any ideas?

I'm using Filbeat 7.2.1, Elasticsearch 7.2.1. Also the default fields.yml is used.

Answer 1

I just found out that date_detection is disabled for filebeat indices by default (Filebeat version 7.2.1). This can be seen here

var (
    // Defaults used in the template
    defaultDateDetection         = false
    ...

Does not look like it can be overridden.

The workaround for this is to use experimental feature append_fields (experimental at least at the time of writing this post. See here for more.) and add the following to the filebeat.yml config

setup.template.overwrite: true
setup.template.append_fields:
- name: logTimestamp
  type: date

This will make sure that the mapping for logTimestamp is date.