Search code examples
pysparkjupyter-notebookspark-avro

Trouble reading avro files in Jupyter notebook using pyspark

I am trying to read an avro file in Jupyter notebook using pyspark. When I read the file i am getting an error.

I have downloaded spark-avro_2.11:4.0.0.jar, i am not sure where in my code I should be inserting the avro package. Any suggestions would be great.

This is an example of the code I am using to read the avro file

df_avro_example = sqlContext.read.format("com.databricks.spark.avro").load("example_file.avro")

This is the error I get

AnalysisException: 'Failed to find data source: com.databricks.spark.avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;'

Solution

  • download the jar to a location and use the following code snippet in your pyspark app

    import os
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars /path/tojar/spark-avro_2.11:4.0.0.jar pyspark-shell'