I am currently trying to upload parquet files from Google Cloud Storage into a table in BigQuery. I keep receiving the error "Unsupported precision and scale values '(35,2)' for field 'X' of type 'FIXED_LEN_BYTE_ARRAY(DECIMAL(35,2))'. The maximum supported precision and scale is '(38,9)."
To me it seems like (35,2) should fit within the precision and scale requirements of (38,9). Why does this datatype from my parquet file not seem to be allowed by Google Cloud? Any workarounds?
This error message is being thrown since the max number of integer digits supported for decimal parquet types is 29; However, in your case, the decimal data has a precision of 35 and a scale of 2 which means the number of integer digits is 33. Based on this, the available workaround is to modify the precision and scale values to match with the parquet file requirements.
You can take a look on the Decimal annotation official documentation that contains the following explanation when using these parquet type fields:
Parquet types with the DECIMAL annotation may have at most a precision of 38 (total number of digits) and at most a scale of 9 (digits to the right of the decimal). The number of integer digits, which is the precision minus the scale, may be at most 29. For example, DECIMAL(38, 9) is supported because the precision is 38 and the scale is 9. In this example, the number of integer digits is 29. DECIMAL(38, 5) is not supported because it has a precision of 38 and a scale of 5. In this example, the number of integer digits is 33.