Result Execution and Presentation when Blackbox Testing

Question

I have developed a Blackbox Test Environment in Python 3.2 for testing a piece of hardware. In this environment I have a TestExecution.py module where I run my tests as follows:

while(True):
    TestWithRestart("Test122")
    TestWithRestart("Test123",keys="invalid_keys.dat")
    TestWithOneComPort("Test200", keys="invalid_keys.dat")
    TestWithTwoComPorts("Test200")
    TestWithTwoComPorts("Test200", ppc_simulation_script="Test200.pcc")
    TestWithNoComPort()
    TestTime("Test500")
    Test600()
    TestWithComPortNoise("Test600")
    TestWithComPortInteruption("Test601")

Each hardware release I test is represented on my PC by its own Test Environment folder. This folder contains logs, keys and a TestExecution.py. Each Test Case has its own results folder and in this folder I have log folders for each execution of the test.

Its also possible that I need to design new tests for a new hardware release. In this case it can take numerous attempts until I get this test to work properly.

With regard to the Pass/Fail status of a test, I do this by manually checking within my log files. The next improvement will be to automate the process of establishing if a test passed or not. I will write separate classes for this. This process will be ongoing.

I'm wondering if I can integrate my environment with Continuous Integration Software with a view to presenting both test execution and/or results in a nice graphical form. It would also be nice to select the tests I wish to execute. What open source software would you recommend?

Thanks,

Barry

Answer 1

Jenkins. For example, you can dump your test results in JUnit XML format and Jenkins will automatically produce nice graphs.

Plugins depend on your needs, of course, but here is a list of the essential plugins plus my favorites (some of them are bundled in the basic package):

• Ant
• A version control integration plugin (like Subversion, depends on what you are using)
• Parameterized Trigger Plugin
• Build Timeout Plugin
• Log Parser Plugin
• Regex Email Plugin
• Artifact Deployer Plugin
• Extended e-mail Plugin

As a Python programmer you will also benefit greatly from Python Jenkins API Wrapper.

In general, however, be careful with plugins: sometimes they are unstable and/or don't function properly. A look at plugin revision history usually can tell you if it is well-maintained.

You may install Jenkins locally on your machine and play with it for a few days before deciding if it fits your needs.