I used logstash to insert data to elastic search and conf file looks like
grok {
match => [
"message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
"message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
]
overwrite => [ "message" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
Now in Elastic Search mapping is done as
{
"response" => "200",
"geoip" => {
"timezone" => "Asia/Kolkata",
"country_code3" => "IN",
"location" => {
"lon" => 80.2833,
"lat" => 13.0833
},
"region_code" => "TN",
"country_name" => "India",
"longitude" => 80.2833,
"city_name" => "Chennai",
"region_name" => "Tamil Nadu",
"latitude" => 13.0833,
"continent_code" => "AS",
"postal_code" => "600073",
"country_code2" => "IN",
"ip" => "122.15.151.189"
},
"timestamp" => "31/May/2019:05:12:22 -0700",
"request" => "/favicon.ico",
"ident" => "-",
"auth" => "-",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36\"",
"referrer" => "\"http://example.com/\"",
"@timestamp" => 2019-05-31T12:12:22.000Z,
"httpversion" => "1.1",
"verb" => "GET"
}
And Now when tried to load make a visualizer in KIBANA geoHash doesn't work as geoip.location is not defined as "type": "geo_point". I have already inserted the whole data. So is there anything I can do to modify the mapping and update the index. what I understood is I need to add an extra field in grok and make "geoip.location" field and reparse the whole log again. but is there any way to create a column in existing Elastic Index and Merge Data from 2 columns?
Like geoip.location2 merged from geoip.location.lon and location.lat
Normally you cannot modify the mapping for fields that are already present in the index
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#_updating_existing_field_mappings
However, you can update the mapping of the index by adding another field. You can check the exact syntax based on the version of Elastic that you're using, but for me on Elastic 6.2 the following worked:
PUT geo_test1/_mapping/geo_test1
{
"properties": {
"geoip": {
"properties": {
"location_geo": {
"type": "geo_point"
}
}
}
}
}
Then, you can use update_by_query and a small painless script to fill the data in the new "location_geo" field based on the data already present in the geoip.location:
POST geo_test1/_update_by_query
{
"script": {
"source": "String lat = ctx._source.geoip.location.lat.toString(); String lon = ctx._source.geoip.location.lon.toString(); ctx._source.geoip.location_geo = lat + ',' + lon",
"lang": "painless"
},
"query": {
"match_all": {}
}
}