Kafka Connect Logstash

Question

How to Connect with Logstash via Apache Kafka? the question how to get it out of Kafka and into something like Elasticsearch inevitably comes up. Does any one has a tuto to do this?

Thank you

Answer 1

Logstash has an input plugin for Kafka. First of all, you should getting familiar with Apache Kafka and its producer/consumer paradigm: https://kafka.apache.org/.

Then getting started with Logstash: https://www.elastic.co/products/logstash. After all of this, you will be able to use Kafka input plugin for logstash: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html.

The last step is build a logstash pipeline to insert data into a destination like Elasticsearch. This simple example can help you to achieve your goal:

logstash.conf

input {  
    kafka {
        bootstrap_servers => "localhost:9092"
        topics => ["example-topic"]
    }
}

output {  
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "example-index"
    }
}

Here we simply take data coming from kafka queue on a specific topic. Then we store the data into an elasticsearch index. Hope that helps!