Search code examples
javaclassnotfoundexceptionapache-storm

Apache Storm : ClassNotFoundException when deploying jar to remoteCluster

I am trying to deploy my topology bundled as a fat jar to remote Cluster . The jar can be deployed successfully but when deployed , I am seeing the following error in the worker log : It says it cannot find class for JpaRepository as shown below

enter image description here

The jar that I am submitting to cluster already contains this class though . I copied jar from the cluster and saw its contents and here is what I see

enter image description here

Would be really thankful if anyone of u has ny idea why is this failing as I am not having any clue how to proceed further on this . I am already deploying jar with class but still it says classNotFound :(( . Everything works well on Local Cluster.

Also One thing :- The Jar I am uploading is of 68MB (bit on heavier side) . Can that have anything to do with this??

Solution

  • If you submit via Eclipse you should to the following:

    public static void main(String[] args) {
    TopologyBuilder b = new TopologyBuilder();
    // build your topology
    b.setSpout(...);
    b.setBolt(...);

    Config c = new Config();
    c.put(Config.NIMBUS_HOST, "130.211.244.139");
    // not sure you you use 6627; 6123 is default port; if you change the port, just use 6627 of course
    c.put(Config.NIMBUS_THRIFT_PORT, new Integer(6123));

    StormSubmitter.submitTopology("myTopolgyName", conf, b.createTopology());
}

    Furthermore, you need to specify JVM argument -Dstorm.jar=/Users/agarg/Documents/notificationRepo/apache-storm/build/libs/apache-storm-SN‌​APSHOT-ns.r134-boot.jar.

    If you want to avoid including transitive dependencies into you jar, you can also copy them manually to Storm's lib folder. Of course, you need to copy them to all machines. You also might need to restart the cluster. You can copy as many jars as you which to lib folder -- Storm will "pick up" all of them.

    Furthermore, if you build a fat jar, the dependent jars cannot be nested into the far jar (ie, the extracted content of dependent jars must be included into the far jar). For example, you dependent jar dep.jar contains a file DClass.class; thus, your fat jar must not contain "dep.jar" (neither in top level nor "lib" folder) but "DClass.class" next to all your Spout and Bolt classes (ie, "DClass.class" must be contained on top level folder within the jar. Of course, you also need to respect package structure, ie, if "dep.jar" contains a file dpackage.DClass2 (ie, "DClass2.class" in folder "dpackage") the far jar must contain a directory "dpackage" (in top level folder within the jar) that contains "DClass2.class".