Logstash: Kafka Output Plugin - Issues with Bootstrap_Server

Question

I am trying to use the logstash-output-kafka in logstash:

Logstash Configuration File

input {
  stdin {}
}

output {
  kafka {
  topic_id => "mytopic"
  bootstrap_server => "[Kafka Hostname]:9092"
  }
}

However, when executing this configuration, I am getting this error:

[ERROR][logstash.agent ] Failed to execute action 
{:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
:exception=>"LogStash::ConfigurationError", :message=>"Something is wrong 
with your configuration."

I tried to change "[Kafka Hostname]:9092" to "localhost:9092", but that also fails to connect to kafka. Only when I remove the bootstrap_server configuration (which then defaults to localhost:9092) then the kafka connection seems to be established.

Is there something wrong with the bootstrap_server configuration of the kafka output plugin? i am using logstash v6.4.1, logstash-output-kafka v7.1.3

Answer 1

I think there is a typo in your configuration. Instead of bootstrap_server you need to define bootstrap_servers.

input {
  stdin {}
}

output {
  kafka {
  topic_id => "mytopic"
  bootstrap_servers => "your_Kafka_host:9092"
  }
}

According to Logstash Docs,

bootstrap_servers
Value type is string
Default value is "localhost:9092"

This is for bootstrapping and the producer will only use it for getting metadata (topics, partitions and replicas). The socket connections for sending the actual data will be established based on the broker information returned in the metadata. The format is host1:port1,host2:port2, and the list can be a subset of brokers or a VIP pointing to a subset of brokers.