Let's start with my setup. Application servers send log files to kinesis data stream, which in turn sends them to kinesis firehose, which eventually send them to elastic search. All the kinesis components are AWS specific tools. In so many words, it means I have little to no control to the settings before elasticsearch. That said, the logs are flowing correctly and reach elasticsearch. So far so good. The problem is that the mapping is totally wrong. All the field are marked as text and no timestamp get recognized. I have prepared a template for the new indexes with the correct mapping:
{
"cms_access-template": {
"order": 0,
"index_patterns": [
"cms_access-*"
],
"settings": {},
"mappings": {
"properties": {
"request": {
"type": "text"
},
"referrer": {
"type": "text"
},
"agent": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"size": {
"type": "integer"
},
"ident": {
"type": "text"
},
"host": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"client": {
"type": "ip"
},
"time": {
"type": "integer"
},
"user": {
"type": "text"
},
"X-Forwarder-For": {
"type": "text"
},
"status": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"timestamp": {
"format": "dd/MMM/yyyy:HH:mm:ss +SSSS",
"type": "date"
}
}
},
"aliases": {}
}
}
As you can see it's a simple apache access log. When I try to create an index pattern based on those index, the timestamp is not recognized and all the field are set as text.
I'm new to elasticsearch and maybe I'm missing something. Are the template used to prepare a mapping for new indexes? If so, how to link them? I searched around and it seems common to send the mapping with the index. But I cannot do that as there's no way in the kinesis tools used in AWS. Can you please help me to create new indexes that will take the mapping from the template?
As mentioned, I'm new to elasticsearch :-) After a lot of head banging I come across a comment in some forum that pointed me in the right direction. It's imperative that when you send the data the template already exist. So, I deleted all the data already sent, I created the template and only then I sent new data. After that the mapping was working as described in the template.