kafka in logstash config and sending the output to ES

Question

I am trying to take the data from Kafka and push it to ES index.

input {
kafka {
bootstrap_servers => "XX.XXX.XXX.XXX:9092"
topics => ["timeseries_ht"]
}
}
output {
elasticsearch { hosts => ["XX.XXX.XXX.XXX:9200"]
index => "sound"
}
}

After running this, index is not created in my ES host . . Is there anything wrong with the configuration? I am using LOGSTASH 5.0.0. and ES 5.0.0 as well.

In Logstash 2.x version :

input {
kafka {
zk_connect => "XX.XXX.XXX.XXX:2181"
topic_id => ["timeseries_ht"]
}
}
output {
elasticsearch {
hosts => ["XX.XXX.XXX.XXX:9200"]
index => "sound"
}
}

Doesnt work with Logstash 2.x as well. I am not able to see the index creation in my host machine.

Can anyone help where I am doing wrong here ?

Answer 1

With Logstash 2 you can configure your kafka input like this and that will work:

input {
  kafka {
    zk_connect => "XX.XXX.XXX.XXX:2181"
    topic_id => ["timeseries_ht"]
    auto_offset_reset => "smallest"
    reset_beginning => true
  }
}
output {
  elasticsearch {
    hosts => ["XX.XXX.XXX.XXX:9200"]
    index => "sound"
  }
}