The Bitbucket pipelines test tab is excluding the "<system-out>" section of my Junit test reports

Question

I have configured my tests per the docs: https://support.atlassian.com/bitbucket-cloud/docs/test-reporting-in-pipelines/

I'm generating test reports with the console output captured though and it appears Bitbucket insists on excluding this output from the test tab.

I can find no documentation from Bitbucket on how this is configurable.

My report looks like this:

<?xml version="1.0" encoding="utf-8"?><testsuites><testsuite name="pytest" errors="0" failures="1" skipped="0" tests="1" time="3.402" timestamp="2023-01-28T20:38:30.862709" hostname="6745580f0e58"><testcase classname="tests.mytest" name="test_force_error" time="0.090"><failure message="ValueError: Some error.">Traceback (most recent call last):
  File "/workspaces/app/tests/mytest.py", line 29, in test_force_error
    assert blah.blah(
  File "/workspaces/app/tests/mytest.py", line 91, in blah
    raise ValueError(
ValueError: Some error.</failure><system-out>--------------------------------- Captured Log ---------------------------------

--------------------------------- Captured Out ---------------------------------
I WANT TO SEE THIS IN THE TEST TAB

</system-out><system-err>--------------------------------- Captured Err ---------------------------------
WARNING: I WANT TO SEE STDERR TOO

</system-err></testcase></testsuite></testsuites>  

I the test tab in the bitbucket UI everything after </failure> is not shown.

How do I have bitbucket include </system-out> and <system-err>? I deliberately included this in the test reports so I could view it in the test tab for each test individually and not have to pour over the entire test output to see it.

Answer 1

The "Tests" tab on the Bitbucket Pipelines web page only shows test failures from your CI build. Unfortunately, it does not display passed tests. The test failures are extracted from the ./**/test-reports/**/*.xml or ./**/test-results/**/*.xml file, which has a stack trace contained inside a pair of <failure> and </failure> tags.

In order to display your custom system-out and system-err messages, you'll need to append those texts to an existing stack trace in the XML file (for a failed test), and ensure that the text is kept inside the <failure> and </failure> tags.

Something like this:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite name="pytest" errors="0" failures="1" skipped="0" tests="1" time="3.402" timestamp="2023-01-28T20:38:30.862709" hostname="6745580f0e58">
    <testcase classname="tests.mytest" name="test_force_error" time="0.090">
      <failure message="ValueError: Some error.">Traceback (most recent call last):
File "/workspaces/app/tests/mytest.py", line 29, in test_force_error
assert blah.blah(
File "/workspaces/app/tests/mytest.py", line 91, in blah
raise ValueError(
ValueError: Some error.

======== system-out ==========
----- Captured Out Logs -------
Console log message 1
Console log message 2

======== system-err ==========
----- Captured Err Logs -------
Console log message 3
Console log message 4
      </failure>
    </testcase>
  </testsuite>
</testsuites>

Important note: The "Tests" tab on the Bitbucket Pipelines only shows the first 8-16 lines of a stack trace for a failed test. I don't know how it decides whether to show 8 or 16 lines, but anything longer than 16 lines will be truncated. So you have to try to reduce the amount of lines of log output, or merge separate lines into long lines.