Subprocess.popen() cannot use quotation marks within arguments on Windows

Question

I went through post after post on SO looking for a method to use quotation marks inside of arguments using subprocess.popen and I cannot seem to find a way.

This works fine from the commandline

runme.bat --include="check|check2"

Python

#!/usr/bin/python
import sys
import subprocess
import shlex

#command_line = "./runme.sh --include=\"check|check2\""
command_line = "runme.bat --include=\"check|check2\""

arg = shlex.shlex(command_line)
arg.quotes = '"'
arg.whitespace_split = True
arg.commenters = ''
command_line_args = list(arg)
print command_line_args

command_line_process = subprocess.Popen(
    command_line_args,
    universal_newlines=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

line = ""
while True:
    line = command_line_process.stdout.readline()
    if line:
        print line
        break

runme.bat

echo %* >> someargs.txt

runme.sh

#!/bin/bash
echo $@

I heard that subprocess.call() is a way around this but I'd like to be able to iterate line by line through the subprocess' output while the program is running.

Edit:

This seems to be a bug in Python because running runme.bat in cmd works correctly, running runme.py in linux works correctly, it's only when running runme.py on Windows where it doesn't work correctly. I created a ticket here.

Edit2:

It's not a python bug apparently lol. Look at chosen answer.

Answer 1

On Windows, a string is a native API. To avoid unnecessary conversions, pass the command as a string:

#!/usr/bin/env python
from __future__ import print_function
import subprocess

command = 'runme.bat --include="check|check2"'
process = subprocess.Popen(command,
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
    universal_newlines=True, bufsize=1)
for line in iter(process.stdout.readline, ''):
    print(line, end='')

stderr=subprocess.STDOUT merges stderr into stdout. If you set stderr=PIPE then you should read from process.stderr in parallel with reading from process.stdout otherwise your program may deadlock.

Popen() passes the string to CreateProcess() Windows function. If the child process is actually a batch-file; you should probably pass shell=True explicitly to make it clear that the command is interpreted using cmd.exe rules (^, |, etc are meta-characters, for more details read the links in this answer).

If you want to pass the argument using %1 instead of %* so that it includes
the whole --include="check|check2" (not only --include) then you could use additional quotes around the argument as @eryksun suggested in the comments:

command = '"runme.bat" "--include="check^^^|check2""'

Notice: triple ^ to escape | here.