Redis, Jedis connection pool optimizeand monitoring tools

I am new to Redis, Jedis. Today, I saw an error in my log

Caused by: redis.clients.jedis.exceptions.JedisExhaustedPoolException: redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_191]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_191]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_191]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:593) ~[?:1.8.0_191]
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677) ~[?:1.8.0_191]
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735) ~[?:1.8.0_191]
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160) ~[?:1.8.0_191]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174) ~[?:1.8.0_191]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:1.8.0_191]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_191]
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583) ~[?:1.8.0_191]
        at com.nokia.snmpapp.VariableDataUtils.processSnmpDataByOP(VariableDataUtils.java:372) ~[snmpapp.jar:?]
       at com.nokia.snmpapp.consumerservice.DocsisQos3MibData.processDocsQos3ParamSetTableObjectEvent(DocsisQos3MibData.java:172) ~[snmpapp.jar:?]
        at com.nokia.snmpapp.KafkaConsumer.onSnmpappDocsQos3ParamSetTableObjectEvent(KafkaConsumer.java:315) ~[snmpapp.jar:?]
        ... 16 more 

Caused by: redis.clients.jedis.exceptions.JedisExhaustedPoolException: Could not get a resource since the pool is exhausted
            at redis.clients.jedis.util.Pool.getResource(Pool.java:53) ~[snmpapp.jar:?]
            at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[snmpapp.jar:?]
            at com.nokia.snmpapp.RedisUtils.add2list(RedisUtils.java:27) ~[snmpapp.jar:?]
            at com.nokia.snmpapp.VariableDataUtils.lambda$processSnmpDataByOP$5(VariableDataUtils.java:379) ~[snmpapp.jar:?]
            at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_191]
            at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_191]
            at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_191]
            at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) ~[?:1.8.0_191]
            at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[?:1.8.0_191]
            at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
            at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040) ~[?:1.8.0_191]
            at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058) ~[?:1.8.0_191]
            at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_191]
            at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_191]  

Caused by: java.util.NoSuchElementException: Timeout waiting for idle object               
           at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:439) ~[snmpapp.jar:?]
           at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349) ~[snmpapp.jar:?]
           at redis.clients.jedis.util.Pool.getResource(Pool.java:50) ~[snmpapp.jar:?]
           at redis.clients.jedis.JedisPool.getResource(JedisPool.java:234) ~[snmpapp.jar:?]
           at com.nokia.snmpapp.RedisUtils.add2list(RedisUtils.java:27) ~[snmpapp.jar:?]
           at com.nokia.snmpapp.VariableDataUtils.lambda$processSnmpDataByOP$5(VariableDataUtils.java:379) ~[snmpapp.jar:?]
           at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[?:1.8.0_191]
           at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) ~[?:1.8.0_191]
           at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_191]
           at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291) ~[?:1.8.0_191]
           at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[?:1.8.0_191]
           at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_191]
           at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040) ~[?:1.8.0_191]
           at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058) ~[?:1.8.0_191]
           at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_191]
           at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_191]

My current configuration is

  maxTotal: 10
  maxIdle: 5
  minIdle: 1
  maxWaitMillis: 2000

I know I can increase my connection pool size to resolve the error message. However, I would like to understand the resource usage a bit more before determining a good number for my env. I have been google about this, but I am unable to find much talking about the monitoring tool or how to monitor the performance and pool usage. Does that mean there is no tools to monitor? If there is no way to monitor connection pool usage, what can I do to get a better idea on what to optimize on Redis connection?

Thanks in advance

Solution

Monitoring Jedis Pool with JMX MBeans

Jedis Pool is based on Apache Generic-Pool API, so you can use JMX to monitor the state of your Connection pool.

For example, start your application with JMX enabled:

-Dcom.sun.management.jmxremote.port=9999 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.ssl=false

Then connect with JConsole or any other JMX compliant tool:

Connect to your application/process
Go to MBeans
Go to the org.apache.commons.pool2.GenericObjectPool.pool MBeans

You will be able to monitor what is happening to your application/JedisPool.

You have many monitoring tools that are compliant with JMX allowing you to put alert based on a threshold

Sizing and Configuration

The configuration of your pool is really application-specific, and to configure the maxTotal property, that is the maximum number of connection you have to look at:

how many conccurent connection is needed by your services
how long it take each call to execute the commands you are sending

And you need to be sure you do not overload your system (client-side and server-side.)

JedisPool: Be sure you release the connection ;)

Also, it is not about monitoring but more about best-practice:

does your application release the connection to the pool after each use?

/// Jedis implements Closeable. Hence, the jedis instance will be auto-closed after the last statement.
try (Jedis jedis = pool.getResource()) {

   // use your jedis
  jedis.set("foo", "bar");

} // the resource will be released back to the pool

If not using the try with the resource you have to close the connection (to put it back to the pool)

Jedis jedis = null;
try {
  jedis = pool.getResource();
  // work with Redis
  jedis.set("foo", "bar");
} finally {
  // you must close the connection to put it back to the pool
  if (jedis != null) {
    jedis.close();
  }
}

Also when you close your application:

 pool.close()