I am trying to a script run.sh using python subprocess
The script creates a csv file and a state.json file
This is the script(works fine when i run it from terminal) :
#!/bin/bash
export GOOGLE_APPLICATION_CREDENTIALS="/home/anti/Documents/bq_s3/BQ/bin/client_secret.json"
/home/anti/Documents/bq_s3/BQ/bin/tap-bigquery -c /home/anti/Documents/bq_s3/BQ/bin/tap_config.json --catalog /home/anti/Documents/bq_s3/BQ/bin/catalog.json --start_datetime '2013-01-01T00:00:00Z' --end_datetime '2013-01-02T00:00:00Z' | /home/anti/Documents/bq_s3/targetcsv/targetcsv/bin/target-csv --config /home/anti/Documents/bq_s3/targetcsv/targetcsv/bin/target_config.json > /home/anti/Documents/bq_s3/state.json
But when i try to run the same from python, it shows the correct output , but doesnt create csv file and a state.json file
This is the python code
#!/usr/bin/python3
import subprocess
from ast import literal_eval
import io
import os
from subprocess import call
shellscript = subprocess.Popen(["/home/anti/Documents/bq_s3/run.sh"], stdout=subprocess.PIPE)
shellscript.wait()
print (shellscript.returncode)
Output
INFO Sending version information to singer.io. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
INFO Running query:
SELECT start_date,trip_seconds,trip_miles FROM `testbq-305217.test_1.tablename` WHERE 1=1 AND datetime '2013-01-01 00:00:00.000000' <= CAST(start_date as datetime) AND CAST(start_date as datetime) < datetime '2013-01-02 00:00:00.000000' ORDER BY start_date LIMIT 100
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"endpoint": "test_bq"}}
INFO Syncing stream:test_bq
0
The return code is 0, the program works but the csv and json files are not being created when ran from python.
I have given necessary permission chmod u+rx run.sh to the script too.
As you mentioned in the comments since the issue was that current working directory is not set properly pass cwd param to the subprocess.Popen. Better yet, replace subprocess.Popen() with a higher-level function, as recommended in its documentation.
#!/usr/bin/python3
import subprocess
shellscript = subprocess.call(["/home/anti/Documents/bq_s3/run.sh"], cwd=YOUR_WORKING_DIR)
print(shellscript)