Snaplogic Writing large amounts of data to BigQuery

Question

I have a pipeline that contains a Pipeline execute snap. The execute snap calls a pipeline that reads data from canvas using postgressql execute snap then writes it to a bigquery database table. The data being written contains over 9,600,000 records. I am using a bigquery write snap. It writes 9619291 out of 9620091 records and completes with the error below. How can I debug this? Is it timing out?

Caused by:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 503 Service Unavailable
{
  "code": 503,
  "errors": [
    {
      "domain": "global",
      "message": "Service is unavailable. Please retry.",
      "reason": "backendError"
    }
  ],
  "message": "Service is unavailable. Please retry.",
  "status": "UNAVAILABLE"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
    at com.snaplogic.snaps.google.bigquery.Write.executeInsert(Write.java:296)
    ... 16 more
Reason: 503 Service Unavailable
POST
    https://bigquery.googleapis.com/bigquery/v2/projects/project_id-/datasets/dataset/tables/wrk_cd2_le_submission_versions/insertAll
{
  "code": 503,
  "errors": [
    {
     
    "domain": "global",
      "message": "Service is unavailable. Please retry.",
      "reason":
    "backendError"
    }
  ],
  "message": "Service is unavailable. Please retry.",
  "status":
    "UNAVAILABLE"
}
Resolution: Verify if the query and other properties are correctly defined.

Answer 1

I was using an execute pipeline snap to call the BigQuery Write snap. I removed it and attached the snap directly to the pipeline generating the data and it ran successfully. I had a similar issue using the Bulk Load (streaming). I noticed it does not work well when called from and Execute pipeline snap.