Search code examples
google-cloud-platformgoogle-bigquerygoogle-cloud-pubsub

Unable to setup GCP Pub/sub subscription to BigQuery when BIGDECIMAL in target table schema

Objective: Setup a GCP Pub/sub subscription:

  • on a topic associated to a protobuf schema
  • which directly routes the JSON-encoded messages to a BigQuery table containing a BIGNUMERIC field

Issue: Creating the topic consistently errors out with a generic API returned error: ‘Request contains an invalid argument.’.

What was attempted: After many combinations of protobuf schema + target table schema, the conversion of any protobuf numeric type to a BigQuery BIGNUMERIC field seems to be the offending aspect.

I found this official doc table which describes the available conversions, and I notice that the target ZetaSQL types do not feature BIGNUMERIC at all.

Question: Am I missing something, or does it actually mean there is no way to do Pub/sub-to-BigQuery ingestion if the target table has a BIGNUMERIC field?

Solution

  • The BIGNUMERIC type is not currently supported in Pub/Sub to BigQuery subscriptions. The only way to ingest into BigQuery from Pub/Sub with this datatype would be to use Dataflow.