Combine two index into third index in elastic search using logstash

Question

I have two index

1. employee_data {"code":1, "name":xyz, "city":"Mumbai" }
2. transaction_data {"code":1, "Month":June", payment:78000 }

I want third index like this 3)join_index

{"code":1, "name":xyz, "city":"Mumbai", "Month":June", payment:78000 } How it's possible??

i am trying in logstash

input {
  elasticsearch {
    hosts => "localost"
    index => "employees_data,transaction_data"
   
     query => '{ "query": { "match": { "code": 1} } }'
    scroll => "5m"
    docinfo => true
  }
}
output {

elasticsearch { hosts => ["localhost"]

index => "join1"
   }

}

Answer 1

You can use elasticsearch input on employees_data

In your filters, use the elasticsearch filter on transaction_data

input {
  elasticsearch {
    hosts => "localost"
    index => "employees_data"
   
     query => '{ "query": { "match_all": { } } }'
     sort => "code:desc"

    scroll => "5m"
    docinfo => true
  }
}
filter {
    elasticsearch {
              hosts => "localhost"
              index => "transaction_data"
              query => "(code:\"%{[code]}\"
              fields => { 
                    "Month" => "Month",
                    "payment" => "payment" 
                   }
        }
}
output {
  elasticsearch { 
    hosts => ["localhost"]
    index => "join1"
   }
}

And send your new document to your third index with the elasticsearch output

You'll have 3 elastic search connection and the result can be a little slow. But it works.