Bash command runs in shell, but not via Python subprocess

Question

I want to run the following Bash command in Python:

import subprocess

command = 'tar -vcz -C /mnt/20TB_raid1  Illumina_Sequencing_Data | pigz | tee >(md5sum > "md5sum_Illumina_Sequencing_Data.txt") | split -b 50G - /mnt/4TB_SSD/Illumina_Sequencing_Data.tar.gz.part-'

subprocess.run(command.split(" "))

When I run this code I get the following error:

tar: 50G: Invalid blocking factor
Try 'tar --help' or 'tar --usage' for more information.

However, when I run this command in the shell directly it run without any issues. It looks like some code is ignored as it thinks the split -b flag is parsed to tar?

Answer 1

It's the piping that causes the problem.

By default the run function runs a single command directly. It does not use a shell to run the command. And piping is a shell thing, it's handled all by the shell itself. Same with redirection and any kind of substitution.

Furthermore, since you split the command in Python, you will pass the pipe signs and all the piped commands as actual arguments to the tar command.

You need to invoke a shell if you want to use piping.