Search code examples
javaeclipsejarmapreducebiginsights

Eclipse Class not found Exception even when jars are included in classpath

Running java mapreduce program on Biginsights server from Eclipse gives com.mysql.jdbc.Driver ClassNotFoundException even though jars are included in classpath

Error :

The system encountered the following error while it was submitting the job:
15/03/26 18:59:29 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/03/26 18:59:29 INFO mapred.JobClient: Cleaning up the staging area hdfs://biginsights-node:9000/user/biadmin/.staging/job_201503261700_0014
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
    at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:164)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1079)
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1099)
    at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:188)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1010)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:963)
    at java.security.AccessController.doPrivileged(AccessController.java:366)
    at javax.security.auth.Subject.doAs(Subject.java:572)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:963)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:616)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:646)
    at example.DbExportMainClass.main(DbExportMainClass.java:51)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
    at java.lang.reflect.Method.invoke(Method.java:619)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
    at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.getConnection(DBInputFormat.java:190)
    at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:158)
    ... 19 more
Caused by: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
    at java.lang.Class.forNameImpl(Native Method)
    at java.lang.Class.forName(Class.java:199)
    at org.apache.hadoop.mapreduce.lib.db.DBConfiguration.getConnection(DBConfiguration.java:148)
    at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.getConnection(DBInputFormat.java:184)
    ... 20 more

Tried almost all suggestions for similar issues available in stackoverflow.

Any suggestion will be helpful.

Solution

  • You need to include jar in Biginsight lib folder to run it as Biginsights Application. Also add path to biginsights-env file.

    default location : /opt/ibm/biginsights/IHC/lib

    To run as java application should be added to classpath.