Search code examples
pythongoogle-bigquerygoogle-cloud-platformavro

BiqQuery streaming inserts using AVRO format


Is there a way to insert data in AVRO format into BigQuery using streaming inserts? If yes, we are using google-cloud-python and it seems this is not supported. Only supported type is json. Am I missing something?

Documentation on BQ streaming inserts doesn't seem to cover data types. I was only able to find AVRO data format mentioned here, but not in a context of streaming inserts.

Is there a way of inserting data in AVRO format using streaming inserts, could you please link me to any example? If it is not possible to insert AVRO data using streaming inserts then what is the recommended way of inserting high volumes of data into BQ using AVRO? Our usage requirements are exceeding the 1000 jobs per table per day limit for load jobs.


Solution

  • No, currently JSON is the only supported format for bigquery streaming API. BigQuery supports AVRO as load job inputs as you mentioned.