I am new to elasticsearch. I have been searching on this for a couple of days but could not find anything related to it. Actually, I want to index log files that are coming from a spooldir through Apache Flume.
I am able to receive data in elasticsearch. But now I want the log data to be split in several fields and then get indexed. Say for example...
"127.0.0.1 - - [18/Mar/2015:20:36:04 -0700] "GET / HTTP/1.1" 404 994"...this is my log data.
Now indexing should happen like...
field 1 is IP address,
field 2 is timestamp,
field 3 is GET or POST method,
field 4 is protocol,
field 5 is status code,
field 6 is time taken
And then I want to visualize these fields using Kibana. By default I can only see Kibana fields like
_source,
_index,
_id,
_type
I want all these fields to be displayed.
Either use the Morphline interceptor in Flume to parse your data or write your own interceptor which does the job.
Logstash also works very well but it lacks the transactional capabilities of Flume.