I am writing a test_examples.py to test the execution of a folder of python examples. Currently I use glob to parse the folder and then use subprocess to execute each python file. The issue is that some of these files are plots and they open a Figure window that halts until the window is closed.
A lot of the questions on this issue offer solutions from within the file, but how could I suppress the output whilst running the file externally without any modification?
What I have done so far is:
import subprocess as sb
import glob
from nose import with_setup
def test_execute():
files = glob.glob("../*.py")
files.sort()
for fl in files:
try:
sb.call(["ipython", "--matplotlib=Qt4", fl])
except:
assert False, "File: %s ran with some errors\n" % (fl)
This kind of works, in that it suppresses the Figures, but it doesn't throw any exceptions (even if the program has an error). I am also not 100% sure what it is doing. Is it appending all of the figures to Qt4 or will the Figure be removed from memory when that script has finished?
Ideally I would like to ideally run each .py file and capture its stdout and stderr, then use the exit condition to report the stderr and fail the tests. Then when I run nosetests it will run the examples folder of programs and check that they all run.
You could force matplotlib to use the Agg backend (which won't open any windows) by inserting the following lines at the top of each source file:
import matplotlib
matplotlib.use('Agg')
Here's a one-liner shell command that will dynamically insert these lines at the top of my_script.py (without modifying the file on disk) before piping the output to the Python interpreter for execution:
~$ sed "1i import matplotlib\nmatplotlib.use('Agg')\n" my_script.py | python
You should be able to make the equivalent call using subprocess, like this:
p1 = sb.Popen(["sed", "1i import matplotlib\nmatplotlib.use('Agg')\n", fl],
stdout=sb.PIPE)
exit_cond = sb.call(["python"], stdin=p1.stdout)
You could capture the stderr and stdout from your scripts by passing the stdout= and stderr= arguments to sb.call(). This would, of course, only work in Unix environments that have the sed utility.
This is actually quite an interesting problem. I thought about it a bit more, and I think this is a more elegant solution (although still a bit of a hack):
#!/usr/bin/python
import sys
import os
import glob
from contextlib import contextmanager
import traceback
set_backend = "import matplotlib\nmatplotlib.use('Agg')\n"
@contextmanager
def redirected_output(new_stdout=None, new_stderr=None):
save_stdout = sys.stdout
save_stderr = sys.stderr
if new_stdout is not None:
sys.stdout = new_stdout
if new_stderr is not None:
sys.stderr = new_stderr
try:
yield None
finally:
sys.stdout = save_stdout
sys.stderr = save_stderr
def run_exectests(test_dir, log_path='exectests.log'):
test_files = glob.glob(os.path.join(test_dir, '*.py'))
test_files.sort()
passed = []
failed = []
with open(log_path, 'w') as f:
with redirected_output(new_stdout=f, new_stderr=f):
for fname in test_files:
print(">> Executing '%s'" % fname)
try:
code = compile(set_backend + open(fname, 'r').read(),
fname, 'exec')
exec(code, {'__name__':'__main__'}, {})
passed.append(fname)
except:
traceback.print_exc()
failed.append(fname)
pass
print ">> Passed %i/%i tests: " %(len(passed), len(test_files))
print "Passed: " + ', '.join(passed)
print "Failed: " + ', '.join(failed)
print "See %s for details" % log_path
return passed, failed
if __name__ == '__main__':
run_exectests(*sys.argv[1:])
Conceptually this is very similar to my previous solution - it works by reading in the test scripts as strings, and prepending them with a couple of lines that will import matplotlib and set the backend to a non-interactive one. The string is then compiled to Python bytecode, then executed. The main advantage is that it this ought to be platform-independent, since sed is not required.
The {'__name__':'__main__'} trick with the globals is necessary if, like me, you tend to write your scripts like this:
def run_me():
...
if __name__ == '__main__':
run_me()
A few points to consider:
set_backend trick won't work and you'll still get figures popping up. The easiest way is to run it directly from the shell (~$ python exectests.py testdir/ logfile.log), or from an (i)python session where you haven't set an interactive backend for matplotlib. It should also work if you run it in a different subprocess from within your ipython session.contextmanager trick from this answer to redirect stdin and stdout to a log file. Note that this isn't threadsafe, but I think it's pretty unusual for scripts to open subprocesses.