Search code examples
javaaerospike

Aerospike Java Async library seems stuck

I am executing a udf using aeropike async client. The client has been initialized as:-

    try {
        asyncClientPolicy.maxThreads = 40;
        asyncClientPolicy.asyncMaxCommands = 20;
        asyncClientPolicy.maxSocketIdle = 13;
        asyncClientPolicy.asyncMaxCommandAction = MaxCommandAction.BLOCK;
        asyncClientPolicy.asyncSelectorThreads = 7;
        asyncClientPolicy.asyncTaskThreadPool = Executors.newFixedThreadPool(20, new ThreadFactory() {
            public final Thread newThread(Runnable runnable) {
                Thread thread = new Thread(runnable);
                thread.setDaemon(true);
                return thread;
            }
        });
    }

So I have 20 threads to handle the callbacks. After a while I see that my program is not doing any progress, indicating deadlock/starvation. Jstack gives me following result:-

java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000064048f338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
    at com.aerospike.client.async.AsyncCluster$BlockBufferQueue.getByteBuffer(AsyncCluster.java:114)
    at com.aerospike.client.async.AsyncCluster.getByteBuffer(AsyncCluster.java:68)
    at com.aerospike.client.async.AsyncCommand.execute(AsyncCommand.java:59)
    at com.aerospike.client.async.AsyncClient.execute(AsyncClient.java:949)
    at main.java.labs.RuleEngineAerospikeConnection.updatePositiveSegmentsUDF(RuleEngineAerospikeConnection.java:217)
    at main.java.labs.Segment$ProductChecker.onSuccess(Segment.java:346)
    at com.aerospike.client.async.AsyncRead.onSuccess(AsyncRead.java:149)
    at com.aerospike.client.async.AsyncCommand.finish(AsyncCommand.java:293)
    at com.aerospike.client.async.AsyncSingleCommand.read(AsyncSingleCommand.java:59)
    at com.aerospike.client.async.AsyncCommand.run(AsyncCommand.java:261)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

I have exactly 20 threads in waiting state all on execute command. Am I doing something wrong with the setup, because I expect the client to return, even if there is any exception in my UDF, what happens if there is an infinite loop? could it lead to this behavior?

Solution

  • It's hard to say what is causing your deadlock without looking at your source code. Your AsyncClientPolicy looks sufficient.

    In any case, the old AsyncClient class has been obsolete since version 4. The AerospikeClient class now includes new async methods which perform must faster than the old AsyncClient. The new async methods also support Netty event loops and always run in non-blocking mode.