Search code examples
azure-pipelineshandlebars.jsowasp

Azure pipline with OWASP ZAP and Handlebars not producing report


So I have this Azure release pipeline as follows

- task: DockerInstaller@0
  displayName: 'Install Docker'

- task: CSE-DevOps.zap-scanner.custom-build-release-task.owaspzap@1
  displayName: 'Run ZAP Scan'
  inputs:
    scantype: targetedScan
    url: '$(targetUrl)'
    port: 443
  continueOnError: true

- bash: |
   npm install -g handlebars-cmd
   cat <<EOF > owaspzap/nunit-template.hbs
   <test-run id="2" name="Owasp test" start-time="{{@generated}}"  >
    <test-suite id="{{@index}}" type="Assembly" name="{{[@name]}}" result="Failed" failed="{{alerts.length}}">
        <attachments>
            <attachment>
                <filePath>owaspzap/report.html</filePath>
            </attachment>
        </attachments>
        <test-case id="{{@index}}" name="{{alert}}" result="Failed" fullname="{{alert}}" time="1">
            <failure>
                <message>
                    <![CDATA[{{{desc}}}]]>
                </message>
                <stack-trace>
                    <![CDATA[Solution:{{{solution}}} Reference: {{{reference}}}
                    instances:{{#each instances}}
                        * {{uri}}
                        - {{method}}
                        {{#if evidence}}- {{{evidence}}}{{/if}}
                    {{/each}}]]>
                </stack-trace>
            </failure>
        </test-case>
    </test-suite>
   </test-run>
   EOF

- bash: 'handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml'
  
- task: PublishTestResults@2
  displayName: 'Publish Test Results'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: 'owaspzap/test-results.xml'
    publishRunAttachments: false
  condition: succeededOrFailed()  

So when I run this pipeline it all seems to pass but when I look at the output for the PublishTestResults task I see this

enter image description here

2023-02-09T22:22:39.8408385Z Publishing test results to test run '1003246'.
2023-02-09T22:22:39.8436588Z TestResults To Publish 1, Test run id:1003246
2023-02-09T22:22:39.8473673Z Test results publishing 1, remaining: 0. Test run id: 1003246
2023-02-09T22:22:39.9290477Z ##[warning]Failed to publish test results: AutomatedTestName must be specified for automated test runs when neither TestPointId nor TestCaseId has been specified..
2023-02-09T22:22:40.0280319Z No test runs are present

I also tried a different handlebars step as below

- bash: |
   sudo npm install -g handlebars-cmd
   cat <<EOF > owaspzap/nunit-template.hbs
   {{#each site}}
   <test-run id="2" name="Owasp test" start-time="{{../[@generated]}}"  >
    <test-suite id="{{@index}}" type="Assembly" name="{{[@name]}}" result="Failed" failed="{{alerts.length}}">
        <attachments>
            <attachment>
                <filePath>owaspzap/report.html</filePath>
            </attachment>
        </attachments>
        {{#each alerts}}
        <test-case id="{{@index}}" name="{{alert}}" result="Failed" fullname="{{alert}}" time="1">
            <failure>
                <message>
                    <![CDATA[{{{desc}}}]]>
                </message>
                <stack-trace>
                    <![CDATA[Solution: {{{solution}}} Reference: {{{reference}}}
                    instances:{{#each instances}}
                        * {{uri}}
                        - {{method}}
                        {{#if evidence}}- {{{evidence}}}{{/if}}
                    {{/each}}]]>
                </stack-trace>
            </failure>
        </test-case>
        {{/each}}
    </test-suite>
   </test-run>
   {{/each}}
   EOF

This also appears to complete but in the output I see this

2023-02-10T08:36:04.1124393Z Result Attachments will be stored in LogStore
2023-02-10T08:36:04.1540352Z Run Attachments will be stored in LogStore
2023-02-10T08:36:04.2252107Z No Result Found to Publish '/home/vsts/work/r1/a/owaspzap/test-results.xml'

Can anyone advise what is wrong with these scripts please

Thanks in advance


Solution

  • This code solved this question but raises another which will be a new ticket

    <test-run id="1" name="TestReport" fullname="TestReport" testcasecount="" result="Passed" total="{$NumberOfItems}" passed="{$NumberOfItems}" failed="{$NumberOfItems}" inconclusive="0" skipped="0" asserts="{$NumberOfItems}" start-time="{$generatedDateTime}" end-time="{$generatedDateTime}" duration="0">
      <command-line>a</command-line>
      <test-suite type="Assembly" id="0-1005" name="TestReport" fullname="TestReport" runstate="Runnable" testcasecount="{$NumberOfItems}" result="Passed" site="Child" start-time="{$generatedDateTime}" end-time="{$generatedDateTime}" duration="0.352610" total="{$NumberOfItems}" passed="0" failed="{$NumberOfItems}" warnings="0" inconclusive="0" skipped="0" asserts="{$NumberOfItems}">
        <attachments>
          <attachment>
            <filePath>
              xml_report.xml
            </filePath>
          </attachment>
        </attachments>
        <for-each select="OWASPZAPReport/site/alerts/alertitem">
        <test-case id="0-1001" name="{name}" fullname="{name}" methodname="Stub" runstate="NotRunnable" seed="400881240" result="Passed" label="Invalid" start-time="{$generatedDateTime}" end-time="{$generatedDateTime}" duration="0" asserts="0">
          <failure>
            <message>
              <value-of select="desc"/>. 
              <value-of select="solution"/>
            </message>
            <stack-trace>
              <for-each select="instances/instance">
                <value-of select="uri"/>, <value-of select="method"/>, <value-of select="param"/>,
              </for-each>
            </stack-trace>
          </failure>
        </test-case>
        </for-each>
      </test-suite>
    </test-run>