Search code examples
continuous-integrationgithub-actionsworkflow

Github actions: merge artifacts after matrix steps


I have a job in Github Actions workflow that runs unit tests and then uploads reports to Jira Xray. The thing is tests step takes quite a while to complete, so I want to split task execution into a few smaller chunks using matrix.

I did it for linting and it works well, however for unit tests I'm struggling with how can I collect and merge all reports together so they can be uploaded after all matrix steps are done.

Here's how current unit tests step looks like

 unit-test:
    runs-on: ubuntu-latest
    needs: setup
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: actions/cache@v3
        with:
          path: ${{ env.CACHE_NODE_MODULES_PATH }}
          key: build-${{ hashFiles('**/package-lock.json') }}
      - run: npx nx affected:test --parallel=3 --base=${{ env.BASE_REF}} --head=HEAD # actual unit tests
      - name: Check file existence #checking whether there're reports at all
        if: success() || failure()
        id: check_files
        uses: andstor/file-existence-action@v1
        with:
      # all reports will be placed in this directory
      # for matrix job reports will be separated between agents, so it's required to merge them
          files: 'reports/**/test-*.xml' 
      - name: Import results to Xray
        if: (success() || failure()) && steps.check_files.outputs.files_exists == 'true' && github.event_name == 'push'
        uses: mikepenz/xray-action@v2
        with:
          username: ${{ secrets.XRAY_CLIENT_ID }}
          password: ${{ secrets.XRAY_CLIENT_SECRET }}
          testFormat: 'junit'
          testPaths: 'reports/**/test-*.xml' # that's where I need to grab all reports
          projectKey: 'MY_KEY'
          combineInSingleTestExec: true

Matrix job for linting looks like this. I would like to do the same for unit tests, but at the same time I want to collect all reports as it works in the job above

linting:
    runs-on: ubuntu-latest
    needs: [setup]
    strategy:
      matrix:
        step: ${{ fromJson(needs.setup.outputs.lint-bins) }} # this will be something like [1,2,3,4]
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - uses: actions/cache@v3
        with:
          path: ${{ env.CACHE_NODE_MODULES_PATH }}
          key: build-${{ hashFiles('**/package-lock.json') }}
      # some nodejs logic to run few jobs, it uses "execSync" from "child_process" to invoke the task
      - run: node scripts/ci-run-many.mjs --target=lint --outputTarget=execute --partNumber=${{ matrix.step }} --base=${{ env.BASE_REF}} --head=HEAD 

Solution

  • Figured it myself

      unit-test:
        runs-on: ubuntu-latest
        needs: [setup]
        strategy:
          fail-fast: false
          matrix:
            step: ${{ fromJson(needs.setup.outputs.unit-test-bins) }}
        steps:
          - uses: actions/checkout@v3
            with:
              fetch-depth: 0
          - uses: actions/cache@v3
            with:
              path: ${{ env.CACHE_NODE_MODULES_PATH }}
              key: build-${{ hashFiles('**/package-lock.json') }}
          - run: node scripts/ci-run-many.mjs --target=test --outputTarget=execute --partNumber=${{ matrix.step }} --base=${{ env.BASE_REF}} --head=HEAD
          - name: Upload reports' artifacts 
            if: success() || failure()
            uses: actions/upload-artifact@v3
            with:
              name: ${{ env.RUN_UNIQUE_ID }}_artifact_${{ matrix.step }}
              if-no-files-found: ignore
              path: reports
              retention-days: 1
    
      process-test-data:
        runs-on: ubuntu-latest
        needs: unit-test
        if: success() || failure()
        steps:
          - uses: actions/checkout@v3
    
          - name: Download reports' artifacts
            uses: actions/download-artifact@v3
            with:
              path: downloaded_artifacts
              
          - name: Place reports' artifacts
            run: rsync -av downloaded_artifacts/*/*/ unit_test_reports/
          - name: Check reports existence
            id: check_files
            uses: andstor/file-existence-action@v1
            with:
              files: 'unit_test_reports/**/test-*.xml'
          - name: Import results to Xray
            run: ls -R unit_test_reports/