Hazelcast: LinkageError "attempted duplicate class definition for name"

I saw this error in our hazelcast cluster.

The code is trying to call executeOnKey() on an IMap. e.g.

    IMap<MyKey, MyCachedClass> myMap = hz.getMap("my-map");
    myMap.executeOnKey(myKey, new EntryProcessor() {
        @Override
        public Object process(Map.Entry entry) {
            return entry.getValue().setItem(item);
        }

        @Override
        public EntryBackupProcessor getBackupProcessor() {
            return null;
        }
    });

...and am getting this exception:

Exception in thread "MYTHREAD" java.lang.LinkageError: loader (instance of com/hazelcast/internal/usercodedeployment/impl/ClassSource): attempted duplicate class definition for name: "com/mycompany/model/Item$ItemEnum"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at com.hazelcast.internal.usercodedeployment.impl.ClassSource.define(ClassSource.java:50)
at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.tryToGetClassFromRemote(ClassLocator.java:163)
at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.handleClassNotFoundException(ClassLocator.java:95)
at com.hazelcast.internal.usercodedeployment.impl.ClassSource.loadClass(ClassSource.java:65)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
at com.hazelcast.internal.usercodedeployment.impl.ClassSource.define(ClassSource.java:50)
at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.tryToGetClassFromRemote(ClassLocator.java:161)
at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.handleClassNotFoundException(ClassLocator.java:95)
at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentService.handleClassNotFoundException(UserCodeDeploymentService.java:89)
at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentClassLoader.loadClass(UserCodeDeploymentClassLoader.java:57)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)
at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:237)
at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:646)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at java.util.ArrayList.readObject(ArrayList.java:797)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:82)
at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:75)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:269)
at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:574)
at com.hazelcast.map.impl.operation.EntryOperation.readInternal(EntryOperation.java:263)
at com.hazelcast.spi.Operation.readData(Operation.java:728)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:160)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:106)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:51)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:323)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:398)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:153)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:123)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:110)
at ------ submitted from ------.(Unknown Source)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolve(InvocationFuture.java:127)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:79)
at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:162)
at com.hazelcast.map.impl.proxy.MapProxySupport.executeOnKeyInternal(MapProxySupport.java:1099)
at com.hazelcast.map.impl.proxy.MapProxyImpl.executeOnKeyInternal(MapProxyImpl.java:109)
at com.hazelcast.map.impl.proxy.MapProxyImpl.executeOnKey(MapProxyImpl.java:757)
at com.mycompany.Processor.java

So to me this seemed like:

The 'executeOnKey(item)' command is being sent to the 'other' node in the cluster where this data is located.
The other node is attempting to deserialize the 'item'.
The other node cannot find the class for the inner enum 'Item$ItemEnum'
The other node makes a request to UserCodeDeploymentService to fetch class definition.
Once the class definition is fetched, and it attempts to create the new class, a LinkageError occurs.

I just cannot understand why it's getting a duplicate class exception, when it first claimed it couldn't find the class. And this error is proving difficult to replicate.

We do have 'userCodeDeployment' set to allow loading of remote class definitions. Our config looks like this:

UserCodeDeploymentConfig ucdConfig = new UserCodeDeploymentConfig();
ucdConfig.setEnabled(true);
ucdConfig.setClassCacheMode(UserCodeDeploymentConfig.ClassCacheMode.ETERNAL);
ucdConfig.setProviderMode(UserCodeDeploymentConfig.ProviderMode.LOCAL_AND_CACHED_CLASSES);
config.setUserCodeDeploymentConfig(ucdConfig);

So it sort of looks like it's doing what I'd expect. i.e. getting a remote class definition because it can't find it locally. Is just a puzzle as to why the exception is occurring.

Solution

Eventually worked this one out.

A simple example.

Assume two Hazelcast nodes. Both have userCodeDeployment switched on. (The ability to share class definitions over the wire.)
Node 1 has MyClass on it's classpath, Node 2 does not.
MyClass definition, note the inner enum (I think the inner enum can be swapped for an inner class and the same thing will happen).

public class MyClass implements Serializable {

    private final InnerEnum enumVal;

    public MyClass(InnerEnum enumVal) {
        this.enumVal = enumVal;
    }

    public enum InnerEnum {
        ONE, TWO, THREE;
    }
}

From Node 1, the following code is run:

hz.getMap("myClassMap").put("myClassKey", new MyClass(MyClass.InnerEnum.ONE));
hz.getMap("myInnerEnumMap").put("myInnerEnumKey", MyClass.InnerEnum.TWO);

Then, on Node 2, the following code is run:

System.out.println(hz.getMap("myInnerEnumMap").get("myInnerEnumKey"));
System.out.println(hz.getMap("myClassMap").get("myClassKey"));

On Node 2, I see the following logged:

[TRACE] ClassLocator Loaded class com.mycompany.MyClass$InnerEnum from Member [myhost]:5701 - cc0b2872-7f5b-484b-a40f-7c7ba1fdc165
TWO
[TRACE] ClassLocator Loaded class com.mycompany.MyClass from Member [myhost]:5701 - cc0b2872-7f5b-484b-a40f-7c7ba1fdc165

Exception in thread "main" java.lang.LinkageError: loader (instance of  com/hazelcast/internal/usercodedeployment/impl/ClassSource): attempted  duplicate class definition for name: "com/mycompany/MyClass$InnerEnum"
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:642)
    at com.hazelcast.internal.usercodedeployment.impl.ClassSource.define(ClassSource.java:50)
    at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.tryToGetClassFromRemote(ClassLocator.java:163)
    at com.hazelcast.internal.usercodedeployment.impl.ClassLocator.handleClassNotFoundException(ClassLocator.java:95)
    at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentService.handleClassNotFoundException(UserCodeDeploymentService.java:89)
    at com.hazelcast.internal.usercodedeployment.UserCodeDeploymentClassLoader.loadClass(UserCodeDeploymentClassLoader.java:57)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at com.hazelcast.nio.ClassLoaderUtil.tryLoadClass(ClassLoaderUtil.java:288)
    at com.hazelcast.nio.ClassLoaderUtil.loadClass(ClassLoaderUtil.java:237)
    at com.hazelcast.nio.IOUtil$ClassLoaderAwareObjectInputStream.resolveClass(IOUtil.java:646)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1868)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:82)
    at com.hazelcast.internal.serialization.impl.JavaDefaultSerializers$JavaSerializer.read(JavaDefaultSerializers.java:75)
    at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:48)
    at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:187)
    at com.hazelcast.map.impl.proxy.MapProxySupport.toObject(MapProxySupport.java:1245)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:120)
    at com.mycompany.main(Node2.java:74)

So:

Node 2 is first trying to fetch and display an instance of MyClass$InnerEnum. It fetches the InnerEnum class definition from Node 1 over hazelcast.
Node 2 then tries to fetch and display an instance of MyClass. It fetches the MyClass class definition from Node 1.
Hazelcast then loops through the inner classes of MyClass (which just contains InnerEnum) and asks the class loader to define them without checking whether they already exist.

See (from Hazelcast 3.11.1) com.hazelcast.internal.usercodedeployment.impl.ClassLocator:160

Map<String, byte[]> innerClassDefinitions = classData.getInnerClassDefinitions();
classSource.define(name, classData.getMainClassDefinition());
for (Map.Entry<String, byte[]> entry : innerClassDefinitions.entrySet()) {
    classSource.define(entry.getKey(), entry.getValue());
}

The easiest solution in our case was to pull InnerEnum out as a top level Enum.