logstash filter how to create two (or above ) outputs for the one input

Question

I am getting via http poller one json

  {
 "id":12345
 "name":"",
 "lastname":"",
 "age":12,
 "address":{"city":"XXXX" , "street":"ZZZZ" }
 }

and I would like this to generate two document in my output :

person :

 {
"id":12345
"name":"",
"lastname":"",
"age":12
  }

address :

 {
"city":"XXXX" , 
"street":"ZZZZ" 
  }

meaning I got one event in the input

in the input phase getting one input :

input {
  http_poller {
    urls => {
      test1 => "http://localhost:8080"
    }
}

in the filter phase I would like to :

1. create person event (tag it as P)
2. create address event (tag it as A)

in the output phase I would like to :

1. send P to P type in ES
2. send A to A type in ES

Answer 1

You can achieve that with the clone filter.

First, you need to install the plugin which is not bundled by default:

bin/logstash-plugin install logstash-filter-clone

Then you can modify your Logstash config like this:

input {
  http_poller {
    urls => {
      test1 => "http://localhost:8080"
    }
    type => "A"
  }
}
filter {
    clone {
        clones => [ "P" ]
        add_tag => [ "P" ]
    }
    if [type] == "P" {
        mutate {
            remove_field => [ "address" ]
        }
    } else {
        mutate {
            add_tag => [ "A" ]
            remove_field => [ "id", "name", "lastname", "age" ]
        }
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        document_type => "%{type}"
    }
}