I have a problem regarding spring DefaultManagedTaskExecutor. We are deploying our SpringBoot 3.2.5 application to websphere Liberty and we are using Liberty ManagedExecutorService as external ExecutorService (allowed using concurrent-3.0 WLP feature), this is its definition:
<featureManager>
<feature>concurrent-3.0</feature>
</featureManager>
<managedExecutorService jndiName="concurrent/appTask">
<concurrencyPolicy max="2" maxQueueSize="5" runIfQueueFull="false" maxWaitForEnqueue="0" />
</managedExecutorService>
As in Spring documentation (https://docs.spring.io/spring-framework/reference/6.0/integration/scheduling.html#scheduling-task-executor-types), DefaultManagedTaskExecutor should be used to link jndi server ExecutorService resource to be used in an application. Our bean definition:
@Bean(name = "appTaskExecutor")
@Primary
TaskExecutor appJndiTaskExecutor() {
DefaultManagedTaskExecutor executor = new DefaultManagedTaskExecutor();
executor.setResourceRef(true);
executor.setJndiName("concurrent/appTask");
return executor;
}
Then we are using appTaskExecutor in an @Async("appTaskExecutor") annotated method which is running individual tasks. So given our managedExecutorService definition after running 7 consecutive tasks, the ExecutorService will be fully occupied and will reject new tasks. In the Spring documentation (same link) it is said that TaskRejectedException will be thrown in that case, so we are catching it and handling the event. This all worked fine when we were running our application on WAS application server using commonJ, but when migrating to Liberty we had to switch to jakarta concurrent. Instead of TaskRejectedException we are getting:
java.lang.IllegalStateException: java.lang.UnsupportedOperationException: isShutdown
at com.ibm.ws.concurrent.internal.ManagedExecutorServiceImpl.isShutdown(ManagedExecutorServiceImpl.java:710) ~[?:?]
at org.springframework.core.task.TaskRejectedException.executorDescription(TaskRejectedException.java:72) ~[spring-core-6.1.6.jar:6.1.6]
at org.springframework.core.task.TaskRejectedException.<init>(TaskRejectedException.java:66) ~[spring-core-6.1.6.jar:6.1.6]
at org.springframework.core.task.support.TaskExecutorAdapter.submit(TaskExecutorAdapter.java:132) ~[spring-core-6.1.6.jar:6.1.6]
at org.springframework.scheduling.concurrent.ConcurrentTaskExecutor$ManagedTaskExecutorAdapter.submit(ConcurrentTaskExecutor.java:221) ~[spring-context-6.1.6.jar:6.1.6]
at org.springframework.scheduling.concurrent.ConcurrentTaskExecutor.submit(ConcurrentTaskExecutor.java:172) ~[spring-context-6.1.6.jar:6.1.6]
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.$sw$original$doSubmit$m32m8m3(AsyncExecutionAspectSupport.java:297) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.$sw$original$doSubmit$m32m8m3$accessor$$sw$vbmd262(AsyncExecutionAspectSupport.java) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport$$sw$auxiliary$1ncn2a1.call(Unknown Source) ~[spring-aop-6.1.6.jar:6.1.6]
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInterWithOverrideArgs.intercept(InstMethodsInterWithOverrideArgs.java:85) ~[skywalking-agent.jar:9.3.0]
at org.springframework.aop.interceptor.AsyncExecutionAspectSupport.doSubmit(AsyncExecutionAspectSupport.java) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.invoke(AsyncExecutionInterceptor.java:127) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) ~[spring-aop-6.1.6.jar:6.1.6]
at org.ourCompany.ourApp.gateway.business.async.TaskJobManager$$SpringCGLIB$$0.processTask(<generated>) ~[ourApp-gateway-business-1.3.1.jar:?]
at org.ourCompany.ourApp.gateway.business.async.TaskQueueServiceBean.processTask(TaskQueueServiceBean.java:41) ~[ourApp-gateway-business-1.3.1.jar:1.3.1]
at jdk.internal.reflect.GeneratedMethodAccessor247.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-6.1.6.jar:6.1.6]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:392) ~[spring-tx-6.1.6.jar:6.1.6]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.6.jar:6.1.6]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) ~[spring-aop-6.1.6.jar:6.1.6]
at org.ourCompany.ourApp.gateway.business.async.TaskQueueServiceBean$$SpringCGLIB$$0.processTask(<generated>) ~[ourApp-gateway-business-1.3.1.jar:?]
at org.ourCompany.ourApp.gateway.business.async.TaskSchedulerBean.processTasks(TaskSchedulerBean.java:30) ~[ourApp-gateway-business-1.3.1.jar:1.3.1]
at jdk.internal.reflect.GeneratedMethodAccessor246.invoke(Unknown Source) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
at org.springframework.scheduling.support.ScheduledMethodRunnable.runInternal(ScheduledMethodRunnable.java:130) ~[spring-context-6.1.6.jar:6.1.6]
at org.springframework.scheduling.support.ScheduledMethodRunnable.lambda$run$2(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.6.jar:6.1.6]
at io.micrometer.observation.Observation.observe(Observation.java:499) ~[micrometer-observation-1.12.5.jar:1.12.5]
at org.springframework.scheduling.support.ScheduledMethodRunnable.$sw$original$run$c8tpsq2(ScheduledMethodRunnable.java:124) ~[spring-context-6.1.6.jar:6.1.6]
at org.springframework.scheduling.support.ScheduledMethodRunnable.$sw$original$run$c8tpsq2$accessor$$sw$p2boiv3(ScheduledMethodRunnable.java) ~[spring-context-6.1.6.jar:6.1.6]
at org.springframework.scheduling.support.ScheduledMethodRunnable$$sw$auxiliary$afreg92.call(Unknown Source) ~[spring-context-6.1.6.jar:6.1.6]
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86) ~[skywalking-agent.jar:9.3.0]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java) ~[spring-context-6.1.6.jar:6.1.6]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-6.1.6.jar:6.1.6]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.lang.UnsupportedOperationException: isShutdown
... 50 more
Which is interesting because it is thrown from ManagedExecutorService implementation in Libery after calling TaskRejectedException.executorDescription() - so the exception is trying to be thrown by Spring. I checked the source code for TaskRejectedException in spring core 6.1.6 (https://github.com/spring-projects/spring-framework/blob/6.1.x/spring-core/src/main/java/org/springframework/core/task/TaskRejectedException.java), here is the problematic method:
private static String executorDescription(Executor executor) {
if (executor instanceof ExecutorService executorService) {
return "ExecutorService in " + (executorService.isShutdown() ? "shutdown" : "active") + " state";
} else {
return executor.toString();
}
}
Problem is in the executorService.isShutdown(). In all jakarta concurrent versions specifications (https://jakarta.ee/specifications/concurrency/3.0/jakarta-concurrency-spec-3.0.pdf) in chapter 3.1.6.1 it is said that:
The lifecycle of a ManagedExecutorService is managed by an application server. All lifecycle operations on the ManagedExecutorService interface will throw a java.lang.IllegalStateException exception. This includes the following methods that are defined in the java.util.concurrent.ExecutorService interface: awaitTermination(), isShutdown(), isTerminated(), shutdown(), and shutdownNow().
So am I tripping or is Spring calling illegal method in case of ExecutorService implementation managed by a server container? Anyway I wanted to ask if someone have expirienced this behaviour before filling issue on Spring, because from what I have found this implementation has been there for some years now and I cannot find anything on it online.
I tried playing with managedExecutorService definition and directly autowiring TaskExecutor and running task by manually submitting it, it all comes down to the same things:
I will create and use my own TaskExecutor instead of a server one.
I will use a server defined ManagedExecutorService using jakarta API instead of Spring.
I will increase the number of tasks that can run i parallel and queue of tasks so the ExecutorService will not get easelly full and I will bend the exeption handling to match the returned exeption (which I dont have to say is really a nonsence).
Either way, I can fix this by the first two solutions, no problem there. But I am just curious if I am missing something, or what exactly should I do if I didnt have a choice and needed to use managedExecutorService defined in Liberty.
I suspect that the current ManagedExecutorServiceImpl initially was only an Executor implementation and now an ExecutorService leading to this error. Or that with WebSphere you would get a proxy with only the Executor exposed (as requested by the DefaultManagedTaskExecutor) and now you get the whole object. I suspect that Spring hasn't adapted to that yet.
To circumvent this what you can do is instead of the DefaultManagedTaskExecutor setup a ConcurrentTaskExecutor. The DefaultManagedTaskExecutor is simply an extension of the ConcurrentTaskExecutor which handles the lookup.
You can do the lookup manually using a JndiObjectFactoryBean and wrap it in a proxy that only exposed the Executor interface.
@Bean(name = "appTaskExecutor")
@Primary
ConcurrentTaskExecutor appJndiTaskExecutor(Executor appManagedTaskExecutor) {
return new ConcurrentTaskExecutor(appManagedTaskExecutor);
}
@Bean(name = "appManagedTaskExecutor")
JndiObjectFactoryBean managedTaskExecutor() {
var factory = new JndiObjectFactoryBean();
factory.setProxyInterface(Executor.class);
factory.setResourceRef(true);
factory.setJndiName("concurrent/appTask");
return factory;
}
Now there will only be a proxy implementing Executor wrapping the actual managed executor service. This will now fail the instanceof check and execute the default logic.