Logstash add an event to an existing document in elasticsearch

Question

I already have a document in Elasticsearch with id (let's say 1). I need to configure Logstash so that an event with the same id gets inserted into Elasticsearch.

Example document where an ID already exists:

{
  "name": "abc"
}

Logstash event:

{
  "address": "new york",
  "mobile no": "xxx"
}

The final result in Elasticsearch:

{
  "name": "abc",
  "address": "new york",
  "mobile no": "xxx"
}

I tried using update script in output plugin:

elasticsearch {
    action => "update"
    hosts => [ "localhost:9200" ]
    index => "details"
    scripted_upsert => true
    document_id => "%{id}"
    script => "ctx._source.name = params.event.get('name')"
}

This allows me to add each field (name, address, etc) but I need to insert the entire json event without specifying each field. How to do that?

Answer 1

I think it's way easier than you think ;)

Elasticsearch supports upserting: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-doc_as_upsert

So combining action => update together with doc_as_upsert => true should do the job:

elasticsearch {
    action => "update"
    hosts => [ "localhost:9200" ]
    index => "details"
    doc_as_upsert => true
    document_id => "%{id}"
}