python subrpocess.Popen with pipe and stdout redirected to a file in the command not working

Question

Here is the problem

Let's consider a file :

printf 'this is \\n difficult' \>test

Now I would like to use python with the following bash command:

grep 'diff' test |gzip \>test2.gz

I have tried the following code which didn't work :

import subprocess

command = \['grep', 'diff', 'test', '|', 'gzip', '\>' 'test2.gz'\]

proc = subprocess.Popen(
  command,
  stdin=subprocess.PIPE,
  stdout=subprocess.PIPE,
  stderr=subprocess.STDOUT,
  encoding='utf8', shell=True)

I have then tried to redirect the output to a file using:

 import subprocess

 command = \['grep', 'diff', 'test', '|', 'gzip'\]

 test2 = open('test2.gz', 'w')

 proc = subprocess.Popen(
    command,
    stdin=subprocess.PIPE,
    stdout=test2,
    stderr=subprocess.STDOUT,
    encoding='utf8', shell=True)

But it didn't work either, so I am a bit clueless on how to proceed to be able to pipe and redirect to a file.

Answer 1

This doesn't make any sense:

command = ['grep', 'diff', 'test', '|', 'gzip']

This is attempting to run the grep command with arguments ["diff", "test", "|", "gzip"] -- which is not what you want -- but when using shell=True you need to pass in a string rather than a list.

If you want to use shell scripting features like io redirection, you need to run a shell script.

import subprocess

command = "grep 'diff' test | gzip"

with open('test2.gz', 'w') as test2:
  proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=test2, stderr=subprocess.STDOUT, encoding='utf8', shell=True)