Search code examples
dockergitlab-cigitlab-ci-runnernewman

Newman report generation works locally but not from CI


I have a GitLab CI job running a series of Postman requests using a custom environment. I'm using Newman to run them alongside the newman-reporter-htmlextra npm plugin to generate a test report.

The job looks like the following:

postman-tests:
  stage: postman-tests
  image: 
    name: wojciechzurek/newman-ci
  before_script:
    - cd ci/tests/postman
    - npm install -g newman-reporter-htmlextra
  script:
    - newman run Non-regression_tests.postman_collection.json -e Tests.postman_environment.json \
      --reporters htmlextra --reporter-htmlextra-export newman-results.html
    - ls -la # Check report generation
  artifacts:
    when: always
    paths:
      - newman-results.html
  allow_failure: true

When I run newman on my mac (newman 4.5.0), the requests and associated tests run properly and the report is generated. However, the job fails and the report is not generated:

 $ newman run Non-regression_tests.postman_collection.json -e Tests.postman_environment.json --reporters htmlextra --reporter-htmlextra-export newman-results.html --color

Uploading artifacts...

WARNING: newman-results.html: no matching files    
ERROR: No files to upload                          
ERROR: Job failed: exit code 1

It seems that the issue may be caused by the testing series in itself rather than the report generation, as the job fails even when I don't generate the report.

I tried different runners: Docker with official newman images, SSH and shell over machines where I had installed newman (version 4.5.6) and the htmlextra reporter beforehand. All fail.
It's interesting to note that the tests series and report generation both succeed when run locally on the machines behind the SSH and shell runners, but they do fail when launched from GitLab CI.

What did I forget/do wrong that prevents the test report generation from GitLab CI?


Solution

  • My .yml for testing, look like this - It's very basic but I've just run it again and it was running fine:

    stages:
        - test
    
    newman_tests:
        stage: test
        image: 
            name: postman/newman_alpine33
            entrypoint: [""]
        script:
            - newman --version
            - npm install -g newman-reporter-htmlextra
            - newman run collection.json -e environment.json --reporters cli,htmlextra --reporter-htmlextra-export testReport.html
        artifacts:
            when: always
            paths:
                - testReport.html
    

    One thing that I do have is entrypoint: [""] in the image block.