We are running version 2.4 & using Spring IgniteSpringBean, SpringTransactionManager & Spring Data repositories for cluster & transactional cache access.
Since we have been having a lot of IgniteClientDisconnectedException related issues, i am writing a manual segmentation resolver (by disabling automatic client reconnection with clientReconnectDisabled set to true) which would detect this condition (using a simple cache query that runs periodically) & initiate a disconnect via IgniteSpringBean#close followed by a reconnect as described here
I had other issues with cache access which i resolved as described here
But the problem is that any time i try to use a spring transaction annotated method it fails creating a transaction with an IllegalStateException mentioning that the grid is in invalid state.
I suspect its again the same issue that i faced earlier (referenced above) with stale references and might need to handle it in the code by updating the kernel reference in SpringTransactionManager. I am trying out that fix & will update if it works.
In the meanwhile if someone has a easier/quicker fix approach/suggestion, please share the same.
Below is the exception,
com.***.dcm.exception.InternalServerErrorException: Error while creating the definition. Message is : Could not create Ignite transaction; nested exception is java.lang.IllegalStateException: Grid is in invalid state to perform this operation. It either not started yet or has already being or have stopped [igniteInstanceName=null, state=STOPPED]
at com.***.dcm.topology.middleware.controller.TopologyServiceControllerHelper.createTopologyDefinition(TopologyServiceControllerHelper.java:124)
at com.***.dcm.topology.middleware.controller.TopologyServiceController.createTopologyDefinition(TopologyServiceController.java:129)
2018-07-24 14:11:00.519 ERROR b731ba99e9b2 --- [nio-7099-exec-1] .d.t.m.c.TopologyServiceControllerHelper : Error while creating the definition. Message is : Could not create Ignite transaction; nested exception is java.lang.IllegalStateException: Grid is in invalid state to perform this operation. It either not started yet or has already being or have stopped [igniteInstanceName=null, state=STOPPED]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
The fix that i mentioned earlier worked. For others that may be facing this issue, the fix involves updating the ignite kernel reference in SpringTransactionManager post the reconnect.
Below are the needed code changes,
In SpringTransactionManager.java,
/**
* @author mlekshma
*
* @param ignite
*/
protected void setResourceFactory(final Ignite ignite) {
this.ignite = ignite;
}
I am using an anonymous inner class of SpringTransactionManager to hack the injection because of ignite-spring integration issues in 2.4 (more on that here),
/**
* @author mlekshma
*
*/
@Configuration
@EnableScheduling
@ComponentScan("com.***.***")
@EnableIgniteRepositories(basePackages={"com.***.***"})
@ImportResource("classpath:ignite-client-conf.xml")
@PropertySource("classpath:ignite-client.properties")
@EnableTransactionManagement
public class IgniteClientConfig {
private static final Logger LOGGER = LoggerFactory.getLogger(IgniteClientConfig.class);
public IgniteClientConfig() {
}
/**
* @return
*/
@Bean
@DependsOn("igniteInstance")
@Primary
@Lazy
public PlatformTransactionManager transactionManager() {
final SpringTransactionManager springTransactionManager = new SpringTransactionManager() {
private boolean isInitialized = false;
@Override
public void afterPropertiesSet() throws Exception {
if (isInitialized) {
if(((IgniteKernal) getResourceFactory()).context().gateway().getState() == GridKernalState.STOPPED) {
IgniteKernal igniteKernal = (IgniteKernal) Ignition.ignite(getIgniteInstanceName());
if(igniteKernal != null) {
setResourceFactory(igniteKernal);
}
}
}
}
/**
* @param event
*/
@EventListener
public void handleContextRefresh(final ContextRefreshedEvent event) throws Exception {
LOGGER.info("Setting up tx support..");
super.afterPropertiesSet();
isInitialized = true;
}
};
// Use default grid client instance created..
springTransactionManager.setTransactionConcurrency(TransactionConcurrency.PESSIMISTIC);
return springTransactionManager;
}
}