Search code examples
pythongoogle-bigquerygoogle-colaboratorypyarrow

ArrowIOError occurred reading from BigQuery with google-cloud-bigquery client library (Python)

I had a function to retrieve a pandas dataframe from a query to BigQuery which worked perfectly for the last couple of months. Today, without any changes, it stopped working in GoogleColab Notebooks throwing this exception:

An exception of type ArrowIOError occurred reading from BigQuery. Arguments: ('Cannot read a negative number of bytes from BufferReader.',)

My code:

def read_from_bigquery_client(bq_client, project_id, sql, curr_func):
  try:
    df = bq_client.query(sql, project=project_id).to_dataframe()
    return df
  except Exception as ex:
    template = "An exception of type {0} occurred reading from BigQuery. Arguments:\n{1!r}\nFunction: {2}"
    message = template.format(type(ex).__name__, ex.args, curr_func)
    print(message)
    return None

Client Auth:

credentials = service_account.Credentials.from_service_account_file(local_cred_filename)
bq_client = bigquery.Client(credentials=credentials,
                            project=credentials.project_id)

The queries I have tried work perfectly when applied directly to BigQuery, plus they worked previously as mentioned above.

Thank you for your help.

Solution

  • The new version (1.26.0) of google-cloud-bigquery Python library has been released on 22nd of July. It may occur issue, that haven't been detected yet. The similar issue with corresponding version has been already reported on Github, where you can follow the updates. Additionally, please report the error you have encountered as well.

    As for now, the workaround for ArrowIOError is to downgrade the version of google-cloud-bigquery library.