Search code examples
apache-sparkavroapache-zeppelinhortonworks-data-platform

Error add spark2 dependencies, avro databricks to zeppelin 0.7.3

It started when I try to run this notes

%spark2

val catDF = spark.read.format("avro").load("/user/dstis/shopee_category_14112019")
      
catDF.show()
catDF.printSchema()

or this alternative

%spark2

val catDF = spark.read.format("com.databricks.spark.avro").load("/user/dstis/shopee_category_14112019")
      
catDF.show()
catDF.printSchema()

both returns error message as below:

org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;

Then I tried to re-config maven at advanced zeppelin-env, zeppelin_env_content as below and restart zeppelin

export ZEPPELIN_INTERPRETER_DEP_MVNREPO="https://repo1.maven.org/maven2/"

Then I add the avro external package to spark2 interperter dependecies as below

avro databrick package

When I save the spark2 interperter configuration, an alert message appear

Error setting properties for interpreter 'spark.spark2': Cannot fetch dependencies for com.databricks:spark-avro_2.11:jar:4.0.0

And when I try tu re-run the spark2 script above it's still give me same error message

Then I try to check the zeppelin log and I found this:

INFO [2020-08-07 05:50:01,496] ({qtp1786364562-929} InterpreterRestApi.java[updateSetting]:138) - Update interpreterSetting 2C4U48MY3_spark2
 INFO [2020-08-07 05:50:01,498] ({qtp1786364562-929} FileSystemConfigStorage.java[call]:98) - Save Interpreter Settings to hdfs://mghdop01.dcdms:8020/user/zeppelin/conf/interpreter.json
ERROR [2020-08-07 05:50:02,768] ({Thread-853} InterpreterSettingManager.java[run]:573) - Error while downloading repos for interpreter group : spark, go to interpreter setting page click on edit and save it again to make this interpreter work properly. : Cannot fetch dependencies for com.databricks:spark-avro_2.11:jar:4.0.0
org.sonatype.aether.RepositoryException: Cannot fetch dependencies for com.databricks:spark-avro_2.11:jar:4.0.0
        at org.apache.zeppelin.dep.DependencyResolver.getArtifactsWithDep(DependencyResolver.java:181)
        at org.apache.zeppelin.dep.DependencyResolver.loadFromMvn(DependencyResolver.java:131)
        at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:79)
        at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:96)
        at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:88)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager$3.run(InterpreterSettingManager.java:565)
 INFO [2020-08-07 05:50:08,758] ({qtp1786364562-934} InterpreterRestApi.java[restartSetting]:181) - Restart interpreterSetting 2C4U48MY3_spark2, msg=
 INFO [2020-08-07 05:50:51,049] ({qtp1786364562-934} NotebookServer.java[sendNote]:711) - New operation from 10.0.77.199 : 54803 : admin : GET_NOTE : 2FG5ZG2T8
 WARN [2020-08-07 05:50:51,051] ({qtp1786364562-934} FileSystemNotebookRepo.java[revisionHistory]:171) - revisionHistory is not implemented for HdfsNotebookRepo
 INFO [2020-08-07 05:50:51,182] ({qtp1786364562-944} InterpreterFactory.java[createInterpretersForNote]:188) - Create interpreter instance spark2 for note 2FG5ZG2T8
 INFO [2020-08-07 05:50:51,183] ({qtp1786364562-944} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.SparkInterpreter 1880223590 created
 INFO [2020-08-07 05:50:51,186] ({qtp1786364562-944} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.SparkSqlInterpreter 2127961876 created
 INFO [2020-08-07 05:50:51,190] ({qtp1786364562-944} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.DepInterpreter 2062124054 created
 INFO [2020-08-07 05:50:51,191] ({qtp1786364562-944} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.PySparkInterpreter 1072370991 created
 INFO [2020-08-07 05:50:51,192] ({qtp1786364562-944} InterpreterFactory.java[createInterpretersForNote]:221) - Interpreter org.apache.zeppelin.spark.SparkRInterpreter 1082046315 created
 INFO [2020-08-07 05:50:54,634] ({pool-2-thread-14} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1596690982396_-221132441 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1767495081
 INFO [2020-08-07 05:50:54,635] ({pool-2-thread-14} Paragraph.java[jobRun]:366) - run paragraph 20200806-121622_1573161163 using spark2 org.apache.zeppelin.interpreter.LazyOpenInterpreter@7011ef66
 INFO [2020-08-07 05:50:54,636] ({pool-2-thread-14} RemoteInterpreterManagedProcess.java[start]:137) - Run interpreter process [/usr/hdp/current/zeppelin-server/bin/interpreter.sh, -d, /usr/hdp/current/zeppelin-server/interpreter/spark, -p, 33438, -l, /usr/hdp/current/zeppelin-server/local-repo/2C4U48MY3_spark2, -g, spark2]
 INFO [2020-08-07 05:50:58,646] ({pool-2-thread-14} RemoteInterpreter.java[init]:248) - Create remote interpreter org.apache.zeppelin.spark.SparkInterpreter
 INFO [2020-08-07 05:50:58,941] ({pool-2-thread-14} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:580) - Push local angular object registry from ZeppelinServer to remote interpreter group 2C4U48MY3_spark2:shared_process
 INFO [2020-08-07 05:50:58,975] ({pool-2-thread-14} RemoteInterpreter.java[init]:248) - Create remote interpreter org.apache.zeppelin.spark.SparkSqlInterpreter
 INFO [2020-08-07 05:50:58,980] ({pool-2-thread-14} RemoteInterpreter.java[init]:248) - Create remote interpreter org.apache.zeppelin.spark.DepInterpreter
 INFO [2020-08-07 05:50:58,984] ({pool-2-thread-14} RemoteInterpreter.java[init]:248) - Create remote interpreter org.apache.zeppelin.spark.PySparkInterpreter
 INFO [2020-08-07 05:50:58,993] ({pool-2-thread-14} RemoteInterpreter.java[init]:248) - Create remote interpreter org.apache.zeppelin.spark.SparkRInterpreter
 WARN [2020-08-07 05:51:32,241] ({pool-2-thread-14} NotebookServer.java[afterStatusChange]:2074) - Job 20200806-121622_1573161163 is finished, status: ERROR, exception: null, result: %text org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;
  at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:630)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:190)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
  ... 47 elided

 INFO [2020-08-07 05:51:32,294] ({pool-2-thread-14} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1596690982396_-221132441 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1767495081

How do I resolve this problem because there are no direct post/thread when I browse it in internet? Is there a way to check if the external package, avro databaricks, is have done downloaded for zeppelin?

NB: HDP 2.6.5 Zeppelin 0.7.3 Spark 2.3.0

-thanks-

Solution

  • We have try to solve it by

    1. First, download spark-avro_2.11-4.0.0.jar (from databricks)
    2. Put it under below directory /usr/hdp//spark2/jars/
    3. Go to zeppelin interpreter settings, find spark2 interpreter
    4. Click edit and find spark.jars section
    5. add path of that jar to spark.jars section
    6. save and restart interpreter
    7. Test it as picture below

    success to read avro file in zeppelin

    or below the script

    spark.sqlContext.sparkContext.hadoopConfiguration.set("avro.mapred.ignore.inputs.without.extension", "false")
val catDF = spark.read.format("com.databricks.spark.avro").load("/sample")

    In first line you can set the extension to false or true based on the file