Search code examples
elasticsearchlogstashlogstash-configuration

Reading and using elasticsearch metada with logstash configuration

I am having a problem to read a dump file from elasticsearch in another system and push it to my elasticsearch using Logstash via file input plugin. My dump file looks like this:

{"_index":"logstash-2018.06.14","_type":"doc","_id":"9Q-9AGQBaaf188t_6DmH","_score":1,"_source":{"offset":124076,"tags":["filebeat_json","beats_input_raw_event","_jsonparsefailure"],...}
{"_index":"logstash-2018.06.14","_type":"doc","_id":"DQ-9AGQBaaf188t_6DqH","_score":1,"_source":{"offset":145573,"tags":["filebeat_json","beats_input_raw_event","_jsonparsefailure"],...}

with my configuration file as follow:

input{
        file{
                path=> "/home/vm01/Documents/log/output.json"
                type=>"log"
                start_position => "beginning"
                sincedb_path=>"/home/vm01/Documents/sincedb_redefined"
                codec => multiline
                {
                        pattern => '^\{'
                        negate => true
                        what => previous
                }
        }
}

filter{
        if [type] == "log"{
                json{
                        source=>"message"
                }
        } 
}

output{
        if [type] == "log"{
                elasticsearch{
                        hosts=>"localhost:9200"
                        index=>"log-%{+YYYY.MM.dd}"
                }
        } 
}

But it gave me error like this:

[WARN ] 2018-07-10 13:13:53.685 [Ruby-0-Thread-18@[main]>worker7: /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.07.10", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x17052ccb>], :response=>{"index"=>{"_index"=>"logstash-2018.07.10", "_type"=>"doc", "_id"=>"gvflg2QB1n75DXFZzVPL", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Field [_type] is a metadata field and cannot be added inside a document. Use the index API request parameters."}}}}

I suspect it is because the dump file already contains all metadata of Elasticsearch from previous VMs and it could not be inserted into the new push. Is there a way for me to use the metadata inside the file rather than the one newly created?

Solution

  • I think you should use elasticdump for ingesting this es dump file into elasticsearch.It will create indices using the metadata present in the input logs or you can even specify the name of the index explicitly.

    Link for elasticdump:-> https://www.npmjs.com/package/elasticdump

    elasticdump is very easy to use and at times proves highly useful.

    In the above case I just needed to use the following command(json_stack.log contains the input logs):->

    elasticdump --input=json_stack.log --output=http://192.168.133.187:9200/

    This will create indices based on the metadata present in the input log:-^

    elasticdump --input=json_stack.log --output=http://192.168.133.187:9200/bhavya

    This will create index with the name bhavya:-^

    This data can also be ingested using logstash, but the simpler and better approach is using elasticdump.If you need help installing elasticdump, I'll let you know the steps but try installing it yourself first.