Search code examples
matrixgithub-actions

Is it possible to guarantee matrix strategy jobs are executed in a specific sequence on GitHub Actions?

Context

I've got a workflow with 2 jobs.

  • The first job generates a list of tasks (with name and type) as output
  • The second job uses 3 different actions, where each action is associated to a task type to be executed.

Minimal example to reproduce

jobs:
  orchestration:
    runs-on: ubuntu-latest
    outputs: 
      tasks: ${{ steps.tasks.outputs.tasks }}
    steps:
      - name: Setup Python
        uses: actions/setup-python@v3
        with:
          python-version: '3.x'

      - name: Build Tasks Array
        id: tasks
        run: |
          import json
          import os
          
          tasks = []

          for i in range(1, 20):
            task_type = "TYPE1" if i % 3 == 1 else ("TYPE2" if i % 3 == 2 else "TYPE3")
            tasks.append({"taskId": f"task{i}", "type": task_type})

          print("Tasks list:", tasks)
          with open(os.environ['GITHUB_OUTPUT'], 'a') as f:
            f.write(f"tasks={json.dumps(tasks)}\n")
        shell: python
  
  process:
    runs-on: ubuntu-latest
    needs: [orchestration]
    strategy:
      matrix:
        tasks: ${{fromJson(needs.orchestration.outputs.tasks)}}
    steps:
      - if: contains( matrix.tasks.type , 'TYPE1')
        uses: owner/action-type1@v1
        with:
          TASK_ID: ${{ matrix.tasks.taskId }}

      - if: contains( matrix.tasks.type , 'TYPE2')
        uses: owner/action-type2@v1
        with:
          TASK_ID: ${{ matrix.tasks.taskId }}

      - if: contains( matrix.tasks.type , 'TYPE3')
        uses: owner/action-type3@v1
        with:
          TASK_ID: ${{ matrix.tasks.taskId }}

Issue

As shared in the example above, I use matrix strategy to run the second job, according to the output from the first job.

However, the order in which the matrix jobs are executed import here, and I would like them to run following the same order as the list generated on the first job: task1 --> task2 --> task3 ...

How can I guarantee matrix strategy jobs are executed in a specific sequence on GitHub Actions?

What I tried

According to the official documentation:

The order of the variables in the matrix determines the order in which the jobs are created.

However, when running in parallel, the strategy matrix order is not respected (task 10 can run before task 1).

Solution

  • After some researches, I found this Github Issue regarding the strategy matrix order being broken, as well as this blog post about sequential deploy.

    As shared in the links above, the solution seems to use the strategy.matrix configuration max-parallel: 1, to force the matrix jobs to run 1 by 1 following the list informed.

    Example:

    strategy:
  matrix:
    stage: ['development', 'integration', 'production']
    # When set to true, GitHub cancels all in-progress jobs if any matrix job fails.
    fail-fast: true
    # The maximum number of jobs that can run simultaneously
  max-parallel: 1

    I've made various tests with up to 100 tasks using the minimal reproducible example in the question, generating logs with timestamp to check the time each job has been running as saving it as artifacts. The final output from the log seems to confirm the order is respected.

    Evidence

    Workflow run:

    enter image description here

    Logs generated:

    Task type: TYPE1; Task name: task1; Timestamp: 2023-09-06 18:23:44
Task type: TYPE2; Task name: task2; Timestamp: 2023-09-06 18:24:23
Task type: TYPE3; Task name: task3; Timestamp: 2023-09-06 18:25:04
Task type: TYPE1; Task name: task4; Timestamp: 2023-09-06 18:25:34
Task type: TYPE2; Task name: task5; Timestamp: 2023-09-06 18:26:11
Task type: TYPE3; Task name: task6; Timestamp: 2023-09-06 18:26:45
Task type: TYPE1; Task name: task7; Timestamp: 2023-09-06 18:27:25
Task type: TYPE2; Task name: task8; Timestamp: 2023-09-06 18:28:01
Task type: TYPE3; Task name: task9; Timestamp: 2023-09-06 18:28:35
Task type: TYPE1; Task name: task10; Timestamp: 2023-09-06 18:29:06
Task type: TYPE2; Task name: task11; Timestamp: 2023-09-06 18:29:41
Task type: TYPE3; Task name: task12; Timestamp: 2023-09-06 18:30:11
Task type: TYPE1; Task name: task13; Timestamp: 2023-09-06 18:30:50
Task type: TYPE2; Task name: task14; Timestamp: 2023-09-06 18:31:28
Task type: TYPE3; Task name: task15; Timestamp: 2023-09-06 18:32:01
Task type: TYPE1; Task name: task16; Timestamp: 2023-09-06 18:32:36
Task type: TYPE2; Task name: task17; Timestamp: 2023-09-06 18:33:14
Task type: TYPE3; Task name: task18; Timestamp: 2023-09-06 18:33:50
Task type: TYPE1; Task name: task19; Timestamp: 2023-09-06 18:34:19
Task type: TYPE2; Task name: task20; Timestamp: 2023-09-06 18:34:55
Task type: TYPE3; Task name: task21; Timestamp: 2023-09-06 18:35:35
Task type: TYPE1; Task name: task22; Timestamp: 2023-09-06 18:36:07
Task type: TYPE2; Task name: task23; Timestamp: 2023-09-06 18:36:41
Task type: TYPE3; Task name: task24; Timestamp: 2023-09-06 18:37:12
Task type: TYPE1; Task name: task25; Timestamp: 2023-09-06 18:37:43
...
Task type: TYPE2; Task name: task98; Timestamp: 2023-09-06 19:21:19
Task type: TYPE3; Task name: task99; Timestamp: 2023-09-06 19:21:51

    Conclusion

    To answer the question, the solution in the second job would be to use

      process:
    runs-on: ubuntu-latest
    needs: [orchestration]
    strategy:
      matrix:
        tasks: ${{fromJson(needs.orchestration.outputs.tasks)}}
        fail-fast: true
      max-parallel: 1