Search code examples
mavenapache-sparkflume

java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils at SimpleApp.main(SimpleApp.java:61)

For a project requirement I was trying to build FlumUtils example with spark present in spark examples. I was able to create the jar file. But while trying to execute it I am getting the following error. Can anybody help me in resolving this?

Error: application failed with exception
java.lang.NoClassDefFoundError: org/apache/spark/streaming/flume/FlumeUtils
        at SimpleApp.main(SimpleApp.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:367)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:77)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.flume.FlumeUtils
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)




  I have included the below dependency as pom file



<dependency> <!-- Spark Flume dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-flume_2.10</artifactId>
  <version>1.2.1</version>
</dependency>
<dependency> <!-- Spark Core dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.10</artifactId>
  <version>1.2.1</version>
</dependency>
<dependency> <!-- Spark Steaming dependency -->
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming_2.10</artifactId>
  <version>1.2.1</version>

And I am running the same with spark-submit --class SimpleApp target/simple-project-1.0.jar

Can any one help me for the same?

Solution

  • If you are running the job using spark-sibmit.. make sure you are providing spark-streaming_2.10 jar using --jars option