I am trying to prepare a pull request for my changes to matplotlib here: https://github.com/shmuller/matplotlib.git. After merging with upstream/master (https://github.com/matplotlib/matplotlib.git), I wanted to find out if I broke anything, so I run the test suite (python tests.py -v -a) on upstream/master. I get:
Ran 4688 tests in 555.109s
FAILED(KNOWNFAIL=330, SKIP=9, errors=197, failures=16)
Now on my merged branch:
Ran 4682 tests in 555.070s
FAILED(KNOWNFAIL=330, SKIP=9, errors=200, failures=18)
Darn! Quite close, but not the same! So I did break something that wasn't broken before. Since there are thousands of tests, and lots of errors and failures to begin with, it does not appear obvious to find out what I broke.
So my question is: What's a good way to find out which tests broke that weren't broken before?
tests.py essentially does:
import nose
nose.main()
so I am hoping for a feature in nose that helps me figure out what I broke, but couldn't find anything in the help (nosetests --help). I obviously could log and diff the whole output, but I'm hoping for a more elegant solution.
Save the logs to two files, A and B. Then use a diff tool like Meld or Emacs' M-x ediff to see the differences.
If you have a guess about what test(s) are relevant to the code you changed, then you could run
nosetests /path/to/test_file.py
Fix the errors relevant to the code you changed, and then see if outputs are identical (by running diff).
If you run
nosetests --with-id
then on subsequent runs, adding the --failed flag will cause nosetests to re-run only the failed tests. That may also help you zero-in on the differences.
nosetests --with-id --failed