What is the simplest way to generate a pickles report out of the test results of behave?

Question

I am writing behavior tests in python with behave. I would like to generate a pickles report. What is the simplest way to achieve that?

Answer 1

There multiple possibilities, each coming with its own advantages and inconvenients.

Pickles supports xUnit, NUnit, and cucumber-json reports, among others. So I need to find a way to get a report from behave in one of those formats. xUnit and NUnit are .NET-specific formats, so they have no relation with python. Behave supports, however, json output. Unfortunately, behave's json output is not the cucumber-json format pickles expects.

Convert JUnit test reports into NUnit reports through an XSLT transformation

Of course, behave supports JUnit and it's possible to convert the JUnit test to NUnit through an XSLT transformation. According to this post, there is a (possibly outdated) XSLT transformation here. Such a transformation can be carried out with Apache Ant, which is great, because I am working on TeamCity which comes with a build type of that kind. In Kotlin DSL, this means I could update my build step with something like

steps {

  [whatever script steps or other stuff]

  ant {
    mode = antScript {
      content = """
        <target name="transform" description="Run functional tests">
          <xslt in="${'$'}{PATH_TO_THE_JUNIT_FILE}" out="${'$'}{PATH_TO_THE_NUNIT_FILE}"  style="junit-to-nunit.xsl" />
        </target>
        """.trimIndent()
    }
    targets = "transform"
  }

}

The problem with that approach is that I need to install a lot of Java stuff on my development PC if I want to test such a script and I am not very fond of Java stuff. Other than that, there is this npm package that could help. It has not a lot of stars, but it might work well. To sum up, I am not very comfortable with XSLTs, therefore I don't like this solution.

Use behave2cucumber

The package name sounds really great, and using it seems very easy. So this was a priori my preferred solution. It is as simple as

pip install behave2cucumber
behave --format=json -o json-report.json /path/to/features
python -m behave2cucumber -i json-report.json -o cucumber-report.json

If that worked, I would've been so happy, because essentially everything is available from pypi, sounds really great. However, all my Background steps were flagged as skipped in my cucumber-report.json output, making that solution a no-go, because it doesn't show the right status of my scenarios.

Write my own custom formatter

Following this github issue, I stumbled upon some explanations and some code (same code here). Here are the steps it took me to make it work:

1. copy / paste the code into file venv/Lib/site-packages/behave/formatter/cucumber_json.py
2. run

behave --format=behave.formatter.cucumber_json:PrettyCucumberJSONFormatter -o cucumber-report.json /path/to/features

And that works like a charm. Because I need this formatter in many projects, my next steps will be to produce a python package that I can install in those projects and reference something like this:

# run_behave.py
import os

from behave.configuration import Configuration
from behave.formatter import _registry
from behave.runner import Runner
from my_formatter_package import PrettyCucumberJSONFormatter

here = os.path.dirname(os.path.abspath(__file__))

_registry.register_as("PrettyCucumberJSONFormatter", PrettyCucumberJSONFormatter)
configuration = Configuration(junit_directory="behave_reports")
configuration.format = ["PrettyCucumberJSONFormatter"]
configuration.stdout_capture = False
configuration.stderr_capture = False
configuration.paths = [os.path.join(here, "features")]
Runner(configuration).run()

And that should do the trick. The inconvenient of that approach is that I need to maintain my own formatter. I don't know what new behave versions will change in the json output in the future and how the cucumber-json schema will evolve with time. Hopefully this formatter will get integrated to behave.