Search code examples
javaapache-stormthrift

Apache Storm (1.2.1) Transport exception Broken pipe

I am getting the below error while submitting a topology in Storm 1.2.1 version which we have upgraded recently.

69001 [pool-7-thread-1] WARN  o.a.s.u.NimbusClient - Ignoring exception while trying to get leader nimbus info from <ip>. will retry with a different seed host.
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
    at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:112) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:73) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:136) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:92) [storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:66) [storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:58) [storm-core-1.2.1.jar:1.2.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
Caused by: java.lang.RuntimeException: org.apache.storm.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
    at org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:104) ~[storm-core-1.2.1.jar:1.2.1]
    ... 16 more

Checking nimbus.log file:

o.a.s.t.s.AbstractNonblockingServer$FrameBuffer Thread-9 [ERROR] Read a frame size of 16777216, which is bigger than the maximum allowable buffer size for ALL connections.

Update

Changed nimbus.thrift.max_buffer_size: 20000000 and topology is not submitting also there's no warn/error logs in client & server.

While killing the nimbus I am getting broken pipe exception because the transfer might be incomplete. I am not sure why this is very slow.

What will be the cause for this issue?

Solution

  • Setting storm.thrift.transport
    "org.apache.storm.security.auth.plain.PlainSaslTransportPlugin"     in both client configuration & storm.yml file solved the issue.

    I was using SimpleTransportPlugin in storm.yml & PlainSaslTransportPlugin in client configuration which caused this issue.