I have an ELK stack setup. When I am performing a query on number fields then it is also matching against string fields. For example, I am sending Load Balancer logs to ELK and if I perform backend_processing_time:>5 on that then it is matching against backend_processing_time with value 0.001 too.
On kibana interface, it is showing that the query is matching string in the request message. I am not able to understand how a query against a number field is matching against a string.
In the dev tools section on kibana i tried to run the same query
GET _search
{
"query": {
"range" : {
"backend_processing_time" : {
"gte" : 50000000000
}
}
}
}
Even with so much backend_processing_time i am getting results. I am not able to understand why this is happening.
I searched on other fields also which are of number type and found that all the queries done on number field are getting matched with string type fields.
I am providing a sample search result which i get for backend_processing_time:>500000000 query. It can be seen in this result that backend_processing_time field is so small but still getting a hit.
{
"_index": "logstash-2017.05.10",
"_type": "prod-quizelb-logs",
"_id": "AVvzYRgL49GPTZAKoDer",
"_score": null,
"_source": {
"backendport": 80,
"received_bytes": 0,
"request": "http://en.meaww.com:80/locales/en.json",
"backend_response": 200,
"verb": "GET",
"message": "2017-05-10T17:19:52.881044Z Prod-ELB 172.68.144.71:34803 10.1.91.253:80 0.000075 0.000606 0.000019 200 200 0 1881 \"GET http://en.meaww.com:80/locales/en.json HTTP/1.1\" \"Mozilla/5.0 (Linux; Android 6.0.1; SM-C900F Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/58.0.3029.83 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/122.0.0.17.71;]\" - -\n",
"type": "prod-quizelb-logs",
"clientport": 34803,
"request_processing_time": 0.000075,
"urihost": "en.meaww.com:80",
"response_processing_time": 0.000019,
"path": "/locales/en.json",
"@timestamp": "2017-05-10T17:21:18.280Z",
"port": "80",
"response": 200,
"bytes": 1881,
"clientip": "172.68.144.71",
"proto": "http",
"@version": "1",
"elb": "Prod-ELB",
"httpversion": "1.1",
"backendip": "10.1.91.253",
"backend_processing_time": 0.000606,
"timestamp": "2017-05-10T17:19:52.881044Z"
},
"fields": {
"@timestamp": [
1494436878280
],
"timestamp": [
1494436792881
]
},
"highlight": {
"backend_processing_time.keyword": [
"@kibana-highlighted-field@6.06E-4@/kibana-highlighted-field@"
],
"request": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@://@kibana-highlighted-field@en.meaww.com@/kibana-highlighted-field@:@kibana-highlighted-field@80@/kibana-highlighted-field@/@kibana-highlighted-field@locales@/kibana-highlighted-field@/@kibana-highlighted-field@en.json@/kibana-highlighted-field@"
],
"elb.keyword": [
"@kibana-highlighted-field@Prod-ELB@/kibana-highlighted-field@"
],
"urihost.keyword": [
"@kibana-highlighted-field@en.meaww.com:80@/kibana-highlighted-field@"
],
"verb": [
"@kibana-highlighted-field@GET@/kibana-highlighted-field@"
],
"request.keyword": [
"@kibana-highlighted-field@http://en.meaww.com:80/locales/en.json@/kibana-highlighted-field@"
],
"type": [
"@kibana-highlighted-field@prod@/kibana-highlighted-field@-@kibana-highlighted-field@quizelb@/kibana-highlighted-field@-@kibana-highlighted-field@logs@/kibana-highlighted-field@"
],
"message": [
"2017-05-10T17:19:@kibana-highlighted-field@52.881044Z@/kibana-highlighted-field@ @kibana-highlighted-field@Prod@/kibana-highlighted-field@-@kibana-highlighted-field@ELB@/kibana-highlighted-field@ 172.68.144.71:34803 10.1.91.253:@kibana-highlighted-field@80@/kibana-highlighted-field@ 0.000075 0.000606 0.000019 200 200 0 1881 \"@kibana-highlighted-field@GET@/kibana-highlighted-field@ @kibana-highlighted-field@http@/kibana-highlighted-field@://@kibana-highlighted-field@en.meaww.com@/kibana-highlighted-field@:@kibana-highlighted-field@80@/kibana-highlighted-field@/@kibana-highlighted-field@locales@/kibana-highlighted-field@/@kibana-highlighted-field@en.json@/kibana-highlighted-field@ @kibana-highlighted-field@HTTP@/kibana-highlighted-field@/1.1\" \"@kibana-highlighted-field@Mozilla@/kibana-highlighted-field@/5.0 (@kibana-highlighted-field@Linux@/kibana-highlighted-field@; @kibana-highlighted-field@Android@/kibana-highlighted-field@ @kibana-highlighted-field@6.0.1@/kibana-highlighted-field@; @kibana-highlighted-field@SM@/kibana-highlighted-field@-@kibana-highlighted-field@C900F@/kibana-highlighted-field@ @kibana-highlighted-field@Build@/kibana-highlighted-field@/@kibana-highlighted-field@MMB29M@/kibana-highlighted-field@; @kibana-highlighted-field@wv@/kibana-highlighted-field@) @kibana-highlighted-field@AppleWebKit@/kibana-highlighted-field@/@kibana-highlighted-field@537.36@/kibana-highlighted-field@ (@kibana-highlighted-field@KHTML@/kibana-highlighted-field@, @kibana-highlighted-field@like@/kibana-highlighted-field@ @kibana-highlighted-field@Gecko@/kibana-highlighted-field@) @kibana-highlighted-field@Version@/kibana-highlighted-field@/4.0 @kibana-highlighted-field@Chrome@/kibana-highlighted-field@/@kibana-highlighted-field@58.0.3029.83@/kibana-highlighted-field@ @kibana-highlighted-field@Mobile@/kibana-highlighted-field@ @kibana-highlighted-field@Safari@/kibana-highlighted-field@/@kibana-highlighted-field@537.36@/kibana-highlighted-field@ [@kibana-highlighted-field@FB_IAB@/kibana-highlighted-field@/@kibana-highlighted-field@FB4A@/kibana-highlighted-field@;@kibana-highlighted-field@FBAV@/kibana-highlighted-field@/122.0.0.17.71;]\" - -\n"
],
"urihost": [
"@kibana-highlighted-field@en.meaww.com@/kibana-highlighted-field@:@kibana-highlighted-field@80@/kibana-highlighted-field@"
],
"path": [
"/@kibana-highlighted-field@locales@/kibana-highlighted-field@/@kibana-highlighted-field@en.json@/kibana-highlighted-field@"
],
"verb.keyword": [
"@kibana-highlighted-field@GET@/kibana-highlighted-field@"
],
"proto.keyword": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@"
],
"port": [
"@kibana-highlighted-field@80@/kibana-highlighted-field@"
],
"type.keyword": [
"@kibana-highlighted-field@prod-quizelb-logs@/kibana-highlighted-field@"
],
"proto": [
"@kibana-highlighted-field@http@/kibana-highlighted-field@"
],
"elb": [
"@kibana-highlighted-field@Prod@/kibana-highlighted-field@-@kibana-highlighted-field@ELB@/kibana-highlighted-field@"
],
"backend_processing_time": [
"@kibana-highlighted-field@6.06E@/kibana-highlighted-field@-4"
],
"port.keyword": [
"@kibana-highlighted-field@80@/kibana-highlighted-field@"
]
},
"sort": [
1494436878280
]
}
EDIT
I got the mapping by running GET /logstash-2017.05.11/_mapping/prod-quizelb-logs query in kibana console.
The mapping which I am getting for backend_processing_time is showing this
"backend_processing_time": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword"
}
}
}
So it seems that this field is of text type thus causing this error to happen.
Now I have another confusion i.e. kibana is showing this as number but elasticsearch is showing this of type text. Also, this is getting mapped dynamically as i never created the mapping on my own. I think that they are getting created by logstash at the time grok filter is applied.
You need to take control of the mapping of those index(indices) so that your field will actually be a number. Otherwise, you will not be sure what kind of field type you'll have there. So, basically you need something like this, either in an index template, or a static mapping all the way:
"backend_processing_time": {
"type": "integer"
}