Infinispan NotSerializableException when spinning up WAR in Wildfly 16

Question

I am trying to run my application on 2 Wildfly 16 nodes running in Standalone mode using the standalone-full-ha.xml configuration. When the 2nd node starts up, the first attempts to distribute/balance the default web cache to the new node.

When it does this, I see the following error message in the log on the first node, and the 2nd node fails to start:

13:45:48,487 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (transport-thread--p18-t8) ISPN000073: Unexpected error while replicating: org.infinispan.commons.marshall.NotSerializableException: org.wildfly.transaction.client.ContextTransactionManager
Caused by: an exception which occurred:
    in field com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.transactionManager
    in object com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired@73962bdb
    in field org.jboss.weld.contexts.SerializableContextualInstanceImpl.instance
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@333ebcb5
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@333ebcb5
    in field java.util.Collections$SynchronizedCollection.c
    in object java.util.Collections$SynchronizedList@333ebcd4
    in field org.jboss.weld.contexts.CreationalContextImpl.dependentInstances
    in object org.jboss.weld.contexts.CreationalContextImpl@4dc7055b
    in field org.jboss.weld.contexts.SerializableContextualInstanceImpl.creationalContext
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@57504e37
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@57504e37

13:45:50,718 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread--p18-t8) Failed to send entries to node node2: org.wildfly.transaction.client.ContextTransactionManager: org.infinispan.commons.marshall.NotSerializableException: org.wildfly.transaction.client.ContextTransactionManager
Caused by: an exception which occurred:
    in field com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorBase.transactionManager
    in object com.arjuna.ats.jta.cdi.transactional.TransactionalInterceptorRequired@73962bdb
    in field org.jboss.weld.contexts.SerializableContextualInstanceImpl.instance
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@333ebcb5
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@333ebcb5
    in field java.util.Collections$SynchronizedCollection.c
    in object java.util.Collections$SynchronizedList@333ebcd4
    in field org.jboss.weld.contexts.CreationalContextImpl.dependentInstances
    in object org.jboss.weld.contexts.CreationalContextImpl@4dc7055b
    in field org.jboss.weld.contexts.SerializableContextualInstanceImpl.creationalContext
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@57504e37
    in object org.jboss.weld.contexts.SerializableContextualInstanceImpl@57504e37

Some other things to note:

• It fails when starting up one of my WARs that is distributed inside an EAR, but not the other 2 WARs that are also deployed in the EAR
• My web.xml is marked as <distributable/> for all 3 WARs
• My other caches seem to replicate just fine

I can't seem to figure out where the ContextTransactionManager is being used, or why it my be serialized to a session cache. I'm assuming it might be in my code somewhere, but I can't figure out where to even start looking. Any help would be appreciated!

Update (05/28/2019): Here are a couple screenshots from the management console showing 2 sessions that get created at startup (we use JSP in our web app, and we have a startup service that runs to pre-compile all of the JSP files):

Answer 1

Update: This ended up being a really simple and somewhat stupid problem in the first place. We had marked a method in the LoggedInUser class with the @javax.transaction.Transactional annotation, which was not necessary in the first place, but was causing the serialization issue.

Just wanted to post an update in case anyone else ran across something similar.