Search code examples
loggingkubernetesapache-kafkafluentd

Fluentd is unable to establish connection with Kafka brokers: Errno::ECONNRESET: Connection reset by peer

I have configured a fluentd sidecar container to send logs to kafka.

Here is my fluent.conf file:

<source>
  @type forward
  bind 127.0.0.1
  port 24224
</source>

<match log.default>
@type kafka2

  # list of seed brokers
  brokers node1:9092, node2:9092, node3:9092
  use_event_time true

  # buffer settings
  buffer_type file
  buffer_path /var/log/td-agent/buffer/kafka
  flush_interval 3s

  # topic settings
  default_topic app_logs

  # data type settings
  <format>
    @type json
  </format>

  # producer settings
  max_send_retries 1
  required_acks -1

  #deubg log
  @log_level debug
</match>

The broker hosts are reachable from the pod where the fluentd sidecar container is running. However, the sidecar container is not able to connect to the brokers. It returns the following log:

2020-10-23 11:41:51 +0000 [info]: starting fluentd-1.10.4 pid=8 ruby="2.5.8"
2020-10-23 11:41:51 +0000 [info]: spawn command to main:  cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
2020-10-23 11:41:55 +0000 [info]: adding match pattern="log.default" type="kafka2"
2020-10-23 11:41:56 +0000 [info]: #0 brokers has been set: ["node1:9092", "node2:9092", "node3:9092"]
2020-10-23 11:41:56 +0000 [info]: adding source type="forward"
2020-10-23 11:41:56 +0000 [info]: #0 starting fluentd worker pid=22 ppid=8 worker=0
2020-10-23 11:41:56 +0000 [debug]: #0 buffer started instance=47608408649900 stage_size=0 queue_size=0
2020-10-23 11:41:56 +0000 [info]: #0 initialized kafka producer: fluentd
2020-10-23 11:41:56 +0000 [info]: #0 listening port port=24224 bind="127.0.0.1"
2020-10-23 11:41:56 +0000 [info]: #0 fluentd worker is now running worker=0
2020-10-23 11:41:57 +0000 [debug]: #0 flush_thread actually running
2020-10-23 11:41:57 +0000 [debug]: #0 enqueue_thread actually running
2020-10-23 11:43:02 +0000 [warn]: #0 Send exception occurred: Could not connect to any of the seed brokers:
- kafka://node1:9092: Connection error Errno::ECONNRESET: Connection reset by peer
- kafka://node2:9092: Connection error Errno::ECONNRESET: Connection reset by peer
- kafka://node3:9092: Connection error Errno::ECONNRESET: Connection reset by peer
2020-10-23 11:43:02 +0000 [warn]: #0 Exception Backtrace : /usr/lib/ruby/gems/2.5.0/gems/ruby-kafka-1.3.0/lib/kafka/cluster.rb:448:in `fetch_cluster_info'

How to get rid of this issue? Could it be an issue from the broker nodes side?

Solution

  • From the fluentd host, try using the https://github.com/edenhill/kafkacat to debug Kafka connectivity/access. I'm sure this is a network/firewall issue, not the fluentd.