Search code examples
continuous-integrationautomated-testsgitlabpowershell-4.0

Gitlab pipeline test stage to fail AND create artifacts anyway

I have a gitlab pipeline running on a windows machine with Windows 7 and powershell 4.0.

The .yaml has the typical 3 stages: build, test and deploy.

For the second stage I want to perform some simple tests that generate a log file which should be available after the test stage finishes.

Here the script section from the test:

script:
  - '$exitCode = (start-process C:\app_versions\app_20181211\bin\app.exe -PassThru -Wait).ExitCode'
  - 'cat .\TestLogs\BasicFunctionsTestPlan.log'
  - 'exit $exitCode'
artifacts:
  paths:
    - .\TestLogs
  expire_in: 1 year

Here I had one problem, after the test run has finished the stage finishes always successfully even if the test themselves failed. Then I had to force the script exit with an error code in case the application tells me that the tests failed.

This caused the second problem: the artifacts link do not get created even they are available (my test produce it anyway).

Probably if I knew how to tell gitlab that the test failed in a more clean way, the artifacts would be available anyway.

I agree that the log file is not an artifact but I would like to keep that file in order to check how the tests have performed, maybe there is a better way to save this file.

Thanks in advance for your help!

EDIT:

Looks like there were more people having the same issue here, maybe it helps understanding better the problem.

Solution

  • I had the same question, but it's easily solved:

    You can use artifacts:when to upload artifacts on job failure or despite the failure.

    artifacts:when

    source: Gitlab CI yaml reference: artifacts:when

    Introduced in GitLab 8.9 and GitLab Runner v1.3.0.

    artifacts:when is used to upload artifacts on job failure or despite the failure.

    artifacts:when can be set to one of the following values:

    1. on_success - upload artifacts only when the job succeeds. This is the default.
    2. on_failure - upload artifacts only when the job fails.
    3. always - upload artifacts regardless of the job status.

    Example:

    To upload artifacts only when job fails:

    job:
  artifacts:
    when: on_failure

    allow_failure

    BTW: you can tell Gitlab CI to continue to the next job after a job failure with allow_failure: true

    source: Gitlab CI yaml Reference: allow_failure

    job1:
  stage: test
  script:
    - execute_script_that_will_fail
  allow_failure: true

    So combined it could look something like:

    job1:
  stage: test
  script:
    - execute_script_that_will_fail
  allow_failure: true
  artifacts:
    when: always # or 'on_failure'
    paths:
    - resulting_artifacts