How to run shell command from python subprocess module in tf.py_function in tf.data pipeline?

Question

I have created tf.data.Dataset pipeline variable to which I pass a python function which is trying to run sox command through python's subprocess module. The code is running fine on cpu with windows os but is not able to run on Google Colab's GPU which has linux os. Here is the code -

from scipy.io import wavfile
import tensorflow as tf

    def tf_func(inp_sample):

        def func(inp_sample,<tf.string object>):
            try:
                fdesc, infile = tempfile.mkstemp(suffix=".wav")
                os.close(fdesc)
                fdesc, outfile = tempfile.mkstemp(suffix=".wav")
                os.close(fdesc)
                wavfile.write(infile,<sample rate>,inp_sample.numpy()) //writes audio file to disk
                arguments = ['sox',infile,outfile,'-q','compand',
                            *DRC_PRESET[<tf.string object>.numpy().decode('utf-8')]] 

                subprocess.check_call(arguments)

            finally:
                os.unlink(infile)
                os.unlink(outfile)

            return tf.convert_to_tensor(inp_sample,dtype=tf.float32)

        drc = np.random.choice(<lis of string>)
        [inp_sample,] = tf.py_function(dynamic_compression, 
                        [inp_sample,tf.constant(drc)],[tf.float32])
        inp_sample.set_shape(target_sample_size) //target_size=<some int>

        return inp_sample
...

inp=tf.data.Dataset.from_tensor_slices(inp) // inp shape eg: (4, 500)
inp = inp.map(tf_dynamic_compression)
for i in inp:
    print(i.numpy())

And the error it throws -

UnknownError: 2 root error(s) found.
  (0) Unknown:   CalledProcessError: Command 'None' died with <Signals.SIGINT: 2>.
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/script_ops.py", line 233, in __call__
    return func(device, token, args)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/script_ops.py", line 122, in __call__
    ret = self._func(*args)

  File "/tmp/tmpn_9q_jxm.py", line 24, in dynamic_compression
    ag__.converted_call(subprocess.check_call, dynamic_compression_scope.callopts, (arguments,), None, dynamic_compression_scope)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/impl/api.py", line 541, in converted_call
    result = converted_f(*effective_args)

  File "/tmp/tmpozf4qyav.py", line 50, in tf__check_call
    ag__.if_stmt(cond_1, if_true_1, if_false_1, get_state_1, set_state_1, (), ())

  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/operators/control_flow.py", line 895, in if_stmt
    return _py_if_stmt(cond, body, orelse)

  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/operators/control_flow.py", line 1004, in _py_if_stmt
    return body() if cond else orelse()

  File "/tmp/tmpozf4qyav.py", line 44, in if_true_1
    raise ag__.converted_call(CalledProcessError, check_call_scope.callopts, (retcode, cmd), None, check_call_scope)

How to solve this problem?

Answer 1

The problem was coming due to sox not getting executed by subprocess module. It has been answered earlier. There are two solutions -

First change the arguments line

arguments = " ".join(['sox',infile,outfile,'-q','compand',*DRC_PRESET[<tf.string object>.numpy().decode('utf-8')]])

and then

1. os.system(arguments)

   OR

2. subprocess.check_call(arguments, shell=True)

For me the second worked.