I want to pass argument to a BigQuery script in shell, here is the example of script I wrote
#!/bin/bash
bq query --use_legacy_sql=false --destination_table=abc --append 'select * from `xyz.INFORMATION_SCHEMA.VIEWS` union all Select * from `def.VIEWS`) where table_name = "$1"'
when I run this script and pass the argument, I do not get any errors but no row is appended to the table. whereas when i specify the table_name as rty that row is appended to the table. What am I missing here?
When you run the script you'll get a prompt like:
Waiting on <BIGQUERY_JOB_ID> ... (0s) Current status: DONE
You can inspect the job in many ways, including the bqtool:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID>
If you have jq installed (sudo apt install jq) you can get just the translated query with:
bq show -j --format=prettyjson <BIGQUERY_JOB_ID> | jq '.configuration.query.query'
which will get you something similar to:
select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name = \"$1\"
As you can see the variable is not correctly escaped so no table matches the WHERE filter. To avoid this you can enclose the query in double quotes and the variable in single ones like this:
#!/bin/bash
bq query \
--use_legacy_sql=false \
--destination_table=xyz.abc \
--append \
"select * from xyz.INFORMATION_SCHEMA.VIEWS where table_name='$1'"
You can get the INFORMATION_SCHEMA.VIEWS: command not found error if using back-ticks. You can omit or escape them with a backslash:
"select * from \`xyz\`.INFORMATION_SCHEMA.VIEWS where table_name='$1'"