Unable to process python subprocess stderr on windows

Question

I am trying to simple file copy operation using dd under Windows. The code is simple. Just open the command using subprocess and read stderr. Then process the stderr to show the progress of the operation. Here is the sample/simple code which I an trying to accomplish:-

import time
import signal
import subprocess
import os

def dd_win():
    windd = "dd.exe"
    in_file = os.path.join("E:", "test-in.iso")
    out_file = os.path.join("E:", "test-out.iso")
    parameter1 = "if=" + in_file
    parameter2 = "of=" + out_file
    parameter3 = "bs=1M"
    parameter4 = "--progress"
    command = [windd, parameter1, parameter2, parameter4]
    print command
    dd_process = subprocess.Popen(command, stderr=subprocess.PIPE, stdout=subprocess.PIPE, shell=True, bufsize=0)

    while dd_process.poll() is None:
        time.sleep(.1)
        line = dd_process.stderr.readline()
        if not line:
            break
        print ">>>>>", line.strip()
    if dd_process.poll() is not None:
        print "Process finished."

dd_win()

Here is wht I got as an output. I could see the stderr but unable to process the transfer bytes to convert in to mega bytes:-

C:\Users\user\Documents>python test.py
['dd.exe', 'if=E:test-in.iso', 'of=E:test-out.iso', '--progress']
>>>>> rawwrite dd for windows version 0.6beta3.
>>>>> Written by John Newbigin <jn@it.swin.edu.au>
>>>>> This program is covered by terms of the GPL Version 2.
>>>>>
175,882,240  # <--- Problem output line. Unable to assign to other variable for progress calculation
Process finished.

I actually do not know where the numbers are coming from. Is it from stderr or buffer? If it is from stderr then ">>>>>" should have be printed before the numbers. I work on Windows 7 and using windows version of dd.

Any help is appreciated.

Answer 1

As discussed in chat, the numbers are coming from stderr. By printing the ascii-indexes of each character in line, we discovered that the final line returned by readline() is \t75,881,728 \r175,882,240 \r\n. It looks like the \r embedded in the middle of this string (which DD outputs) is confusing your code.