How can I run a single Terraform GitHub Actions pipeline that applies multiple terraform resources (each with own state files) at same time?

Question

I have a pipeline in GitHub Actions that runs a terraform Plan when a user raises a PR, and an Apply once that PR is approved.

The pipeline is configured to set a Terraform working directory and run Terraform init for a specific directory only e.g user-1/projectA is hard coded into my pipeline

However the same repo also contains additional folders, e.g projectB with the same structure as above. And also additional user folders, e.g user-2;

user-project1 
 └ projectA 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
 └ projectB 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
user-project2 
 └ projectA 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 
 └ projectB 
  └ main.tf 
  └ backend.tf 
  └ variables.tfvars 

I'd like a single pipeline that will run for any change in this monorepo that contains multiple projects, e.g if any change is made in any of the files above.

The context is, I have a monorepo that contains multiple GCP projects. Only I have access to this repo and to make changes. When I get a request from a a bunch of users to make some changes across a number of projects (enabling a new API for example), I'd like to make all those changes and then run a single pipeline to get them all applied.

Each project has its own state file in a remote backend

I am using Terraform CLI only with no option to use TF Cloud or Enterprise

Answer 1

You could use a reusable workflow along with a matrix strategy.

In that case, you would have 2 workflows:

• a main workflow with 2 jobs
  • one that identify the updated folders
  • one that call the reusable workflow for the updated folders.
• a reusable workflow that would receive a folder as input, and then cd to this folder before performing a terraform apply command.

The matrix would be filled with the list of updated folders on the main workflow, calling the reusable workflow for each folder updated.

Here is an example:

main workflow

on:
  push:
    paths:
      - 'folder1/**'
      - 'folder2/**'

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  setup:
    runs-on: ubuntu-latest
    outputs: 
      folders: ${{ steps.array.outputs.array }}
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: changes
        with:
          filters: |
            folder1:
              - 'folder1/**'
            folder2:
              - 'folder2/**'

      - name: Build Array
        id: array
        run: |
          myArray=()
          if [ "${{ steps.changes.outputs.folder1 }}" = "true" ]
          then
            myArray+=("folder1")
          fi
          if [ "${{ steps.changes.outputs.folder2 }}" = "true" ]
          then
            myArray+=("folder2")
          fi
          myArray=$(jq --compact-output --null-input '$ARGS.positional' --args -- "${myArray[@]}")
          echo "Updated folder list: $myArray"
          echo "array=$myArray" >> $GITHUB_OUTPUT
        shell: bash

  build:
    needs: [setup]
    strategy:
      fail-fast: false
      matrix:
        folder: ${{ fromJSON(needs.setup.outputs.folders) }}
    uses: ./.github/workflows/reusable.yml
    with:
      path: ${{ matrix.folder }}

Note: You may use a script instead of doing all the shell commands, but I found it easier that way as the dorny/paths-filter action generates an output for each folder.

reusable workflow

on:
  workflow_call:
    inputs:
      path:
        required: true
        type: string

concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.path }}
    cancel-in-progress: true

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - run: |
          cd ${{ inputs.path }}
          terraform apply