Search code examples
apache-sparkapache-spark-sqluser-defined-functionsapache-zeppelin

Adding JARs to use sql UDF in spark

I want to use custom UDF which are defined in my libaray. I have used the following code for that:

%spark2
import org.apache.spark.sql.functions.year

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

sqlContext.sql(s"ADD JAR /usr/hdp/current/spark-client/lib/myLib.jar")

val df = sqlContext.sql("select parse_datetime(start_timestamp, 'CET', 'yyyyMMddHHmmssSSS') AS TIMESTAMP) from temp)

The above code complains about missing function "parse_datetime", so apparently, ADD JAR statement is not adding myLib. Is there a syntax problem or what is that I am missing. I am running this in Zeppelin.

I have also copied the myLib.jar in $SPARK_HOME/lib folder but no use.

Solution

  • I have found the solution. I have given the exact path of myLib.jar in the dependencies section of spark intertreper. It works now.