Storm 2.0 worker is failing with an exception at start without storm-kafka dependency on pom

Question

As per storm 2.0 release document, storm-kafka is removed and users have to move to storm-kafka-client.

So, I removed storm-kafka dependency from by pom. Already we had storm-kafka-client dependency on pom. Build is successful.

But when starting the topology to storm with storm jar CLI command, we are getting the following exception. I can also see json-simple-1.1.jar in the storm lib folder and we shouldn't be getting NoClassDef exception.

java.lang.Error: java.lang.NoClassDefFoundError: org/json/simple/parser/ParseException
    at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:647) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.utils.Utils.handleUncaughtException(Utils.java:626) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.utils.Utils.lambda$createDefaultUncaughtExceptionHandler$2(Utils.java:982) ~[storm-client-2.0.0.jar:2.0.0]
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057) [?:1.8.0_191]
    at java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052) [?:1.8.0_191]
    at java.lang.Thread.dispatchUncaughtException(Thread.java:1959) [?:1.8.0_191]
Caused by: java.lang.NoClassDefFoundError: org/json/simple/parser/ParseException
    at java.lang.Class.getDeclaredFields0(Native Method) ~[?:1.8.0_191]
    at java.lang.Class.privateGetDeclaredFields(Class.java:2583) ~[?:1.8.0_191]
    at java.lang.Class.getDeclaredField(Class.java:2068) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1857) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494) ~[?:1.8.0_191]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_191]
    at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:542) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:915) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.Task.mkTaskObject(Task.java:271) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.Task.<init>(Task.java:88) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.executor.Executor.mkExecutor(Executor.java:194) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.worker.Worker.loadWorker(Worker.java:218) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.worker.Worker.lambda$start$0(Worker.java:166) ~[storm-client-2.0.0.jar:2.0.0]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
    at org.apache.storm.daemon.worker.Worker.start(Worker.java:165) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.worker.Worker.main(Worker.java:129) ~[storm-client-2.0.0.jar:2.0.0]
Caused by: java.lang.ClassNotFoundException: org.json.simple.parser.ParseException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_191]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_191]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_191]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191]
    at java.lang.Class.getDeclaredFields0(Native Method) ~[?:1.8.0_191]
    at java.lang.Class.privateGetDeclaredFields(Class.java:2583) ~[?:1.8.0_191]
    at java.lang.Class.getDeclaredField(Class.java:2068) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1857) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494) ~[?:1.8.0_191]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391) ~[?:1.8.0_191]
    at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) ~[?:1.8.0_191]
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_191]
    at org.apache.storm.utils.Utils.javaDeserialize(Utils.java:542) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.utils.Utils.getSetComponentObject(Utils.java:915) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.Task.mkTaskObject(Task.java:271) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.Task.<init>(Task.java:88) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.executor.Executor.mkExecutor(Executor.java:194) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.worker.Worker.loadWorker(Worker.java:218) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.worker.Worker.lambda$start$0(Worker.java:166) ~[storm-client-2.0.0.jar:2.0.0]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
    at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_191]
    at org.apache.storm.daemon.worker.Worker.start(Worker.java:165) ~[storm-client-2.0.0.jar:2.0.0]
    at org.apache.storm.daemon.worker.Worker.main(Worker.java:129) ~[storm-client-2.0.0.jar:2.0.0]```

Answer 1

The lib directory is not for client code. If you need json-simple in your topology, include it in your jar, or put it into the worker-lib directory.

Here's the relevant code from the storm.py bootstrap script:

def get_classpath(extrajars, daemon=True, client=False):
    ret = get_wildcard_dir(STORM_DIR)
    if client:
        ret.extend(get_wildcard_dir(STORM_WORKER_LIB_DIR))
    else :
        ret.extend(get_wildcard_dir(STORM_LIB_DIR))
    ...