Search code examples
apache-sparkgoogle-bigqueryavro

Loading Avro into BigQuery via the Spark Connector


From what I've seen in these example, it's only do-able via Gson. Is it possible to directly load Avro objects into a BigQuery table via the Spark Connector? Converting from Avro to BigQuery Json becomes a pain when the avro specification starts going beyond simple primitive values. (e.g. Unions)

Cheers


Solution

  • Not through Spark Connector, but BigQuery supports loading AVRO files directly: https://cloud.google.com/bigquery/loading-data#loading_avro_files