Search code examples
google-bigquerydecimalparquet

How to load a parquet file with a dicimal field into BigQuery?

We have a table named parq store as parquet, and there is a field which is Decimal in parq.

When I load parq into BigQuery, I got a message :

"Parquet type 'FIXED_LEN_BYTE_ARRAY' (DECIMAL) for field 'field_name' is not supported."

Load result:

Waiting on bqjob_r70dd180dddb0942d_00000160bb2ff706_1 ... (1s) Current status: DONE Error in query string: Error processing job 'project_id:bqjob_r70dd180dddb0942d_00000160bb2ff706_1': Parquet type 'FIXED_LEN_BYTE_ARRAY' (DECIMAL) for field 'field_name' is not supported.

Command:

bq load --source_format=PARQUET --time_partitioning_type=DAY "dataset.table$20171001" "gs://data.0.parq"

How can I load parquet file like data.0.parq into Bigquery?

Thanks.

Solution

  • DECIMAL type is not yet supported by BigQuery. According to Google, should be ready in Q1: https://issuetracker.google.com/issues/35906014

    So meanwhile I guess you would have to Cast this field into a Float before storing it into Parquet