Search code examples
hadoopcompressionsqoopsnappy

Error: Could not initialize class org.xerial.snappy.Snappy

I am importing data from RDBMS to hdfs using sqoop 1.4.6 using snappyCodec as compression and avro as file format. I have following components installed

Hadoop 2.8.0
Spark 2.1.0
Hive 1.2.2
Scala 2.11.8
Cassandra 3.10

When i try to import mysql table to hdfs using with following command: 

sqoop import --connect jdbc:mysql://******:****/retail_db --username **** --password **** --table order_items --compress --compression-codec org.apache.hadoop.io.compress.SnappyCodec --target-dir /user/test/order_items --as-avrodatafile

I'm getting following error: 

Could not initialize class org.xerial.snappy.Snappy

and it times out after trying to find this class.
I tried solutions mentioned regarding this issue like mounting tmp and adding snappy jar file ( which is already present ) but none of them works. Anything I am missing ?

Solution

  • After trying several solutions, i finally figured out the problem. Using --verbose flag with sqoop command showed that there were multiple snappy-xxx.jar files being imported from different components like

    1. $HADOOP_HOME/share/hadoop/common/lib/
    2. $HADOOP_HOME/share/hadoop/mapreduce/lib/
    3. $SQOOP_HOME/lib/
      so there were 3 snappy jars being included.

    Solution: I made sure that only 1 file gets included in this case and error was resolved.