In Python 2.7 I have the following code inside certain loop
file = open("log.txt", 'a+')
last_position = file.tell()
subprocess.Popen(["os_command_producing_error"], stderr = file)
file.seek(last_position)
error = file.read()
print(error) # example of some action with the error
The intention is that the error that was just given by stderr gets, say printed, while file is keeping the whole record.
I am a beginner in Python and I am not clear what happens in the stderr = file.
My problem is that error keeps being empty, even though errors keep getting logged in the file.
Could someone explain why?
I have tried adding closing and opening the file again, or file.flush() right after the subprocess line. But still the same effect.
Edit: The code in the answer below makes sense to me and it seems to work for for the author of that post. For me (in Windows) it is not working. It gives an empty err and an empty file log.txt. If I run it line by line (e.g. debugging) it does work. How to understand and solve this problem?
Edit: I changed the Popen with call and now it works. I guess call waits for the subprocess to finish in order to continue with the script.
error is empty because you are reading too soon before the process has a chance to write anything to the file. Popen() starts a new process; it does not wait for it to finish.
call() is equivalent to Popen().wait() that does wait for the child process to exit that is why you should see non-empty error in this case (if the subprocess does write anything to stderr).
#!/usr/bin/env python
import subprocess
with open("log.txt", 'a+') as file:
subprocess.check_call(["os_command_producing_error"], stderr=file)
error = file.read()
print(error)
You should be careful with mixing buffered (.read()) and unbuffered I/O (subprocess).
You don't need the external file here, to read the error:
#!/usr/bin/env python
import subprocess
error = subprocess.check_output(["os_command_producing_error"],
stderr=subprocess.STDOUT)
print(error)
It merges stderr and stdout and returns the output.
If you don't want to capture stdout then to get only stderr, you could use Popen.communicate():
#!/usr/bin/env python
import subprocess
p = subprocess.Popen(["os_command_producing_error"], stderr=subprocess.PIPE)
error = p.communicate()[1]
print(error)
You could both capture stderr and append it to a file:
#!/usr/bin/env python
import subprocess
error = bytearray()
p = subprocess.Popen(["os_command_producing_error"],
stderr=subprocess.PIPE, bufsize=1)
with p.stderr as pipe, open('log.txt', 'ab') as file:
for line in iter(pipe.readline, b''):
error += line
file.write(line)
p.wait()
print(error)
See Python: read streaming input from subprocess.communicate().