Search code examples
google-bigquerygoogle-cloud-pubsub

Troubleshooting Pub/Sub to BigQuery subscription


I set up a BigQuery subscription to a Pub/Sub topic using the table schema, but when I publish a message to the topic, the data is not written to the BigQuery table. I followed Google's troubleshooting advice, but it says the subscription is healthy and there are no error messages. I also tried the advice in this answer, but the table is set to never expire, and enabling ack messages for the subscription has no effect since it is not a pull subscription. The project does have a billing account (we have been using BigQuery for a while now).

Looking at the BigQuery job history, I see other jobs (Datastream) and my queries from the console, but nothing from Pub/Sub.

Looking at the Pub/Sub metrics, I see the oldest unacked message age increases linearly, ack message count has no data, and unacked messages by region and delivery metrics both show the number of messages I published to the topic. So apparently the messages are not being acked.

What else can I do to find out what the problem is? Does Pub/Sub have logs somewhere?


Solution

  • I was able to find the problem by configuring a dead-letter topic on the BigQuery subscription as described here. The messages in the dead-letter topic have an attribute CloudPubSubDeadLetterSourceDeliveryErrorMessage with the error message. (In my case it was a type mismatch in one of the columns.)