Python outputs CR CR LF?

Question

I am in the process of writing a testing script that will simply run an *.EXE file with a few arguments, and then output the result to a file. I have one *.sh testing script that correctly runs the tests (but requires manual updating for more tests). The lines in this script look like this:

blah.exe arg1 arg2 arg3 > ../test/arg4/arg4.out

I have written a python script to automatically generate the arguments based on a few simple python modules (it's pretty rough right now):

import os
import subprocess

for test_dir in os.listdir():
    if not os.path.isdir(test_dir):
        continue
    # load a few variables from the ./test_dir/test_dir.py module
    test_module = getattr(__import__(test_dir, fromlist=[test_dir]), test_dir)

    # These arguments are relative paths, fix the paths
    arg1 = os.path.join(test_dir, test_module.ARG1)
    arg2 = os.path.join(test_dir, test_module.ARG2)
    arg3 = os.path.join(test_dir, test_module.ARG3)

    proc = subprocess.Popen(["../bin/blah.exe", arg1, arg2, arg3], stdout=subprocess.PIPE)

    stdout_txt = proc.stdout.read().decode("utf-8")

    with open(os.path.join(test_dir, test_dir + '.out'), 'w') as f:
        f.write(stdout_txt)

I have opened the files to compare and while the script is working, I ran across a problem. The first (shell) solution outputs correct line-endings. The second (python) solution outputs lines ending with CR CR LF. This looks correct in Notepad, but in Notepad++, every other line appears to be blank:

Why is the output from python giving improper line endings?

How can I correct the line endings? (change CR CR LF to CR LF without having to write another script)

Answer 1

Try using the universal_newlines=True argument to Popen (see http://docs.python.org/library/subprocess.html#popen-constructor).