Why does dumping Dataframe to Avro file fail to convert bytearray in Python?

Question

I face the following difficulty : I am using Spark 1.4.1, Python 2.7.8, and spark-avro_2.10-1.0.0

I am trying to store Python byte-arrays in an avro file using spark-avro. My purpose is to store chains of bytes corresponding to chunks of images that have been encoded using a specific image encoder.

It fails on a conversion exception :

org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.UnresolvedUnionException: Not in union ["bytes","null"]:

Here is a sample example I have made for reproducing the problem :

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, Row

import os
import tempfile


# Just setting name of the Spark app
conf = SparkConf().setAppName("pyspark test")
sc = SparkContext(conf=conf)

sqlContext = SQLContext(sc)

# build Data frame containing bytearrays (stupid)
data = map(lambda x: bytearray(str(x)), range(5))

rdd = sc.parallelize(data)

# convert data to SQL Row
rdd_row = rdd.map(lambda b:  Row(val=b))

# create a DataFrame
df = sqlContext.createDataFrame(rdd_row)
df.registerTempTable('test')

# try to dump it
outputFile = os.path.join(tempfile.gettempdir(), 'test.avro')
df.write.format("com.databricks.spark.avro").save(outputFile)

This is launched using

spark-submit --master local[1] --jars "spark-avro_2.10-1.0.0.jar" testBytearray.py

And it fails in the conversion !

Answer 1

I was using a bad version of spark-avro. After building the latest, everything works fine.