Number of file descriptors increasing infinitely using fluent and flume

I run fluent agent with fluent-plugin-flume inside Docker container. After a period of time some programs inside my container failed with next error:

Check failed: _s.ok() Bad status: Runtime error: Could not create thread: Resource temporarily unavailable (error 11)

I discovered that number of file descriptors inside Docker container increasing infinitely:

sudo lsof| grep flume | wc -l
469292

And after a while number of descriptors becomes bigger then a file descriptors limit. Looks like the thrift connection is handled in a wrong way: file descriptors are still present after the end of transaction.

Fluend config:

<source>
  @type forward
  tag forward_1
  bind 0.0.0.0
  port 24224
  linger_timeout 0
</source>

<match forward_1>
      @type copy
      <store>
              @type flume
              timeout 15
              host localhost
             port 33333
      </store>
</match>

Fume config:

forward_1.sources  = source1
forward_1.channels = channel1
forward_1.sinks = sink1
#
forward_1.sources.source1.type = thrift
forward_1.sources.source1.bind = localhost
forward_1.sources.source1.port = 33333
forward_1.sources.source1.channels = channel1
#
forward_1.channels.channel1.type = memory
forward_1.channels.channel1.capacity = 10000
forward_1.channels.channel1.transactionCapacity = 1000
#
forward_1.sinks.sink1.type = org.apache.kudu.flume.sink.KuduSink
forward_1.sinks.sink1.channel = channel1
forward_1.sinks.sink1.masterAddresses = 10.1.0.1:7051
forward_1.sinks.sink1.tableName = shop_logs
forward_1.sinks.sink1.batchSize = 50
forward_1.sinks.sink1.producer = KuduJsonProducer

Docker version: 17.03 Fluentd version: 0.14.19 Flume version: 1.7.0

Is there a way to limit file descriptors number?

Solution

try add 'threads' in sources to your flume config. like this:

forward_1.sources.source1.threads = number_of_threads