Search code examples
javaspringspring-bootredisbroken-pipe

spring boot redis operation throw broken pipe error


We use redis in spring boot project. After running a period of time, the redis operation MAY throw a broken pipe error, but sometimes it will succeed. Restarting the service will resolve this problem, but it's not a good idea.

I can't tell the reason why it happens. It seems that some redis connections in the pool are unusable, but aren't closed and evicted from the pool.

my questions are:

  • possible reason causing the broken pipe error?
  • if there isn't redis operation for a long period, will the idle connection in the pool become unusable?
  • will the connection be closed and evicted from pool when broken pipe error happen?

pom.xml:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

application.yml:

spring:
  redis:
    database: 0
    host: ${REDIS_HOST:127.0.0.1}
    password: ${REDIS_PASSWORD:password}
    port: ${REDIS_PORT:6379}
    timeout: ${REDIS_TIMEOUT:1000}
    pool:
      max-active: ${REDIS_MAX_ACTIVE:100}
      max-wait: ${REDIS_MAX_WAIT:500}
      max-idle: ${REDIS_MAX_IDLE:20}
      min-idle: ${REDIS_MIN_IDLE:5}

error message:

org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketException: Broken pipe (Write failed); nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe (Write failed)
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:67) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisExceptionConverter.convert(JedisExceptionConverter.java:41) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:37) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:37) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisConnection.convertJedisAccessException(JedisConnection.java:212) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisConnection.hSet(JedisConnection.java:2810) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.DefaultHashOperations$9.doInRedis(DefaultHashOperations.java:173) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:204) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:166) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:88) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    at org.springframework.data.redis.core.DefaultHashOperations.put(DefaultHashOperations.java:170) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketException: Broken pipe (Write failed)
    at redis.clients.jedis.Connection.flush(Connection.java:291) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.jedis.Connection.getIntegerReply(Connection.java:220) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.jedis.BinaryJedis.hset(BinaryJedis.java:749) ~[jedis-2.8.2.jar!/:na]
    at org.springframework.data.redis.connection.jedis.JedisConnection.hSet(JedisConnection.java:2808) ~[spring-data-redis-1.7.6.RELEASE.jar!/:na]
    ... 115 common frames omitted
Caused by: java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_111]
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_111]
    at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_111]
    at redis.clients.util.RedisOutputStream.flushBuffer(RedisOutputStream.java:52) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.util.RedisOutputStream.flush(RedisOutputStream.java:216) ~[jedis-2.8.2.jar!/:na]
    at redis.clients.jedis.Connection.flush(Connection.java:288) ~[jedis-2.8.2.jar!/:na]
    ... 118 common frames omitted

Solution

  • This can happen for n number of reasons, one of them could be when you use a long-living connection (e. g. connect to Redis on application start and then use the connection over and over).

    Some of the things to do are:

    1. Reconnect if the connection is broken (needs some try/catch magic to prevent that errors are propagated to your application logic) or a better was is to use
      TestOnBorrow - Sends a PING request when you ask for the resource.
      TestOnReturn - Sends a PING when you return a resource to the pool.
      TestWhileIdle - Sends periodic PINGS from idle resources in the pool.
    2. Connect at the moment you need the connection and disconnect afterwards

    Regarding
    if there isn't redis operation for a long period, will the idle connection in the pool become unusable?

    maxidle means at any given time the system allows 'maxIdle' many number of connections to be idle the rest will be constantly checked, closed and returned to the pool. I dont know a reason when an idle connection be unusable. Anyways this can be got rid of by using ways mentioned above.