Converting a bash script into Python using subprocess module

Question

I am trying to convert a bash script into a subprocess so I can schedule different parameters. Here is my original bash script:

#!/bin/sh

set -xe

export NVIDIA_VISIBLE_DEVICES=0
export CUDA_VISIBLE_DEVICES=0
export TF_CUDNN_RESET_RND_GEN_STATE=1

python3 -u DeepSpeech.py \
  --train_files /external_data/data_csvs/train.csv \
  --test_files  /external_data/data_csvs/test.csv \
  --dev_files  /external_data/data_csvs/dev.csv \
  --epochs 30 \
  --train_batch_size 32 \
  --dev_batch_size 32 \
  --test_batch_size 32 \
  --export_dir /external_data/deepspeech_models/ \
  --use_allow_growth  \
  --n_hidden 2048 \
  --train_cudnn  \
  --learning_rate 0.00005 \
  --dropout_rate 0.40 \
  --summary_dir /external_data/tensorboard_summaries/ \
  --checkpoint_dir /external_data/mozilla_release_chkpts/deepspeech-0.7.4-checkpoint/ | tee //tmp/external/deepspeech_models/progress.txt \
  "$@"

Now I am trying to convert this into a subprocess with the following:

    subprocess.Popen([
        'set', '-xe',
        'export', 'NVIDIA_VISIBLE_DEVICES=0',
        'export', 'CUDA_VISIBLE_DEVICES=0',
        'export', 'TF_CUDNN_RESET_RND_GEN_STATE=1',
        'python3', '-u', 'DeepSpeech.py',
        '--train_files', '/external_data/data_csvs/train.csv',
        '--test_files'  '/external_data/data_csvs/test.csv',
        '--dev_files', '/external_data/data_csvs/dev.csv',
        '--epochs', str(epochs),
        '--train_batch_size', str(trainbs),
        '--dev_batch_size', str(devbs),
        '--test_batch_size', str(testbs),
        '--export_dir', '/external_data/deepspeech_models/',
        '--use_allow_growth',  
        '--n_hidden', str(2048),
        '--train_cudnn',  
        '--learning_rate', str(lr),
        '--summary_dir', '/external_data/tensorboard_summaries/' 
        '--checkpoint_dir', '/external_data/mozilla_release_chkpts/deepspeech-0.7.4-checkpoint/', '|', 'tee', '/external_data/deepspeech_models/Deepspeech_progress.txt',
        '$@'], shell = True, cwd = '//DeepSpeech/', stdout = subprocess.PIPE, executable = '/bin/sh')

the str() values are just values I am using as variables into my subprocess.

It runs without error but nothing happens. Am I missing something? The bash script config runs fine. Also, how would I go about getting output processed to stdout when I run the script?

Answer 1

As in this answer, you need to separate the individual commands with ;.

In your case, try something like:

train_cmd = ['python3', '-u', 'DeepSpeech.py',
            '--train_files', '/external_data/data_csvs/train.csv',
            '--test_files'  '/external_data/data_csvs/test.csv',
            '--dev_files', '/external_data/data_csvs/dev.csv',
            '--epochs', str(epochs),
            '--train_batch_size', str(trainbs),
            '--dev_batch_size', str(devbs),
            '--test_batch_size', str(testbs),
            '--export_dir', '/external_data/deepspeech_models/',
            '--use_allow_growth',  
            '--n_hidden', str(2048),
            '--train_cudnn',  
            '--learning_rate', str(lr),
            '--summary_dir', '/external_data/tensorboard_summaries/' 
            '--checkpoint_dir', '/external_data/mozilla_release_chkpts/deepspeech-0.7.4-checkpoint/', '|', 'tee', '/external_data/deepspeech_models/Deepspeech_progress.txt',
            '$@']
cmds = ['set -xe', 
        'export NVIDIA_VISIBLE_DEVICES=0', 
        'export CUDA_VISIBLE_DEVICES=0',
        'export TF_CUDNN_RESET_RND_GEN_STATE=1',
        ' '.join(train_cmd)]
subprocess.Popen('; '.join(cmds), shell = True, cwd = '//DeepSpeech/', stdout = subprocess.PIPE, executable = '/bin/sh')