Search code examples
google-bigqueryavrosnappy

Is it possible to load Avro files with Snappy compression into BigQuery?

I know that BigQuery supports Avro file upload and I'm successful in loading Avro file into BigQuery.

Using below command,

java -jar avro-tools-1.7.7.jar fromjson --codec snappy --schema-file SourceSchema.avsc Source.json > Output.snappy.avro

I have generated an Avro file with Snappy compression and trying to load into BigQuery but Load job fails with below errors,

Errors:
file-00000000: The Apache Avro library failed to parse file file-00000000. (error code: invalid)

Is it possible to load Avro files with Snappy compression into BigQuery?

Solution

  • BigQuery only supports DEFLATE and Snappy algorithm for Avro data blocks compression, from the docs (https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro)

    Compressed Avro files are not supported, but compressed data blocks are. BigQuery supports the DEFLATE and Snappy codecs.