Search code examples
pythongoogle-bigquerydatasetconfiggcloud

Google Cloud - BiqQuery, Invalid schema update. Field has changed type from INTEGER to STRING


In GCP, using python, my query selects some data to be inserted in table BQ_table.

query produces 2 fileds/columns: A String and Integer.

BQ_table follows schema.

    schema = [  bigquery.SchemaField("name", "STRING", mode="NULLABLE"),
                bigquery.SchemaField("age", "INTEGER", mode="NULLABLE") 
          ]
    config = bigquery.QueryJobConfig(destination= BQ_table,  autodetect=True, schema=schema)
    query_job = self._client.query(query, job_config=config)
    que

My questions is how can I list/show all possible Datasets and Tables in my current project? I could not find the proper gcloud | gsutil command.

The Data can be inserted, type of age from STRING to INTEGER, but I'm wondering what am I missing to insert the numeric data into the table.

The error is:

Google Cloud - BiqQuery, Invalid schema update. Field has changed type from INTEGER to STRING


Solution

  • List of all datasets in Google BigQuery can be queried from INFORMATION_SCHEMA.SCHEMATA:

    SELECT * FROM region-us.INFORMATION_SCHEMA.SCHEMATA;
    

    List of tables in Google BigQuery can be queried from INFORMATION_SCHEMA.TABLES:

    -- Returns metadata for tables in a single dataset.
    SELECT * FROM myDataset.INFORMATION_SCHEMA.TABLES;
    
    -- Returns metadata for tables in a region.
    SELECT * FROM region-us.INFORMATION_SCHEMA.TABLES;
    

    To insert STRING to INTEGER try to explicitly convert it in query with CAST or SAFE_CAST first:

    SELECT CAST("123" AS INT64);
    SELECT SAFE_CAST("apple" AS INT64);