Showing test count in buildbot

Question

I am not particularly happy about the stats that Buildbot provides. I understand that it is for building and not testing - that's why it has a concept of Steps, but no concept of Test. Still there are many cases when you need test statistics from build results. For example when comparing skipped and failed tests on different platforms to estimate the impact of a change.

So, what is needed to make Buildbot display test count in results?
What is the most simple way, so that a person who don't know anything about Buildbot can do this in 15 minutes?

Answer 1

Depending how you want to process the test results and how the test results are presented, Buildbot does provide a Test step, buildbot.steps.shell.Test

An example of how I use it for my build environment:

from buildbot.steps import shell

class CustomStepResult(shell.Test):
    description = 'Analyzing results'
    descriptionDone = 'Results analyzed'

    def __init__(self, log_file = None, *args, **kwargs):
        self._log_file = log_file
        shell.Test.__init__(self, *args, **kwargs)
        self.addFactoryArguments(log_file = log_file)

    def start(self):
        if not os.path.exists(self._log_file):
            self.finished(results.FAILURE)
            self.step_status.setText('TestResult XML file not found !')
        else:
            import xml.etree.ElementTree as etree
            tree = etree.parse(self._log_file)
            root = tree.getroot()

            passing = len(root.findall('./testsuite/testcase/success'))
            skipped = len(root.findall('./testsuite/testcase/skip'))
            fails = len(root.findall('./testsuite/error')) + len(root.findall('./testsuite/testcase/error')) + len(root.findall('./testsuite/testcase/failure'))

            self.setTestResults(total = fails+passing+skipped, failed = fails, passed = passing)
            ## the final status for WARNINGS is green but the step itself will be orange
            self.finished(results.SUCCESS if fails == 0 else results.WARNINGS)
            self.step_status.setText(self.describe(True))

And in the configuration factory I create a step as below:

factory.addStep(CustomStepResult(log_file = log_file))

Basically I override the default Test shell step and pass a custom XML file which contains my test results. I then look for the pass/fail/skip result nodes and accordingly display the results in the waterfall.