Search code examples
pythongoogle-bigquerypython-bigquery

Bigquery data not getting inserted

I'm using python client library to insert data to big query table. The code is as follows.

client = bigquery.Client(project_id)
errors = client.insert_rows_json(table=tablename,json_rows=data_to_insert)
assert errors == []

There are no errors, but the data is also not getting inserted.

Sample JSON rows:

[{'a':'b','c':'d'},{'a':'f','q':'r'},.....}]

What's the problem? No exception also

Solution

  • got the answer to my question. The problem was I was inserting one more column data for which data was not there. I found a hack in order to find out if the data is not inserting to bigquery table.

    1. Change the data to newline delimited json with the keys as the column names and values as values you want for that particular column.
    2. bq --location=US load --source_format=NEWLINE_DELIMITED_JSON dataset.tablename newline_delimited_json_file.json. Run this command in you terminal and see if throws any errors. If it throws an error it's likely that something is wrong with your data/table schema.
    3. Change the data/table schema as per the error and retry inserting the same via python.

    It's better if the python API throws an error/exception like on the terminal, it would be helpful.