How to block PR from beeing merged if external test result fails

Question

Every time a new PullRequest is created to merge branchA to branchB, I need to run some tests is a external tool. It might take hours to run it.

When the PR is created, a workflow dispatches a github action that creates a EXTERNAL JOB in the external tool. Long story short, it is the only way this external tool works.

So, what is my problem? I need a way to block the PR while the external test is running AND get notified when a final result is available.

A naive approach would be :

• PR is created, a github action starts running
• The github action send a POST to the external tool, creating a job there
• The github action waits (pooling every X seconds) to verify if we have a result

I know that check-suites might be a way to go, but it requires creating a Github App.

Is there a better way to do it?

Answer 1

The simplest way is probably to use a commit status. You need the commit hash of the latest commit in the PR, and then you can set statuses; for example, when your long running test has started, something like (using the GitHub CLI)

gh api repos/{owner}/{repo}/statuses/COMMITSHA \
    -f state='pending' \
    -f context='External job' \
    -f description='Running tests'

You can then make that a required check, so while your external tests run, the PR has the check pending like

and later, when the tests have finished, either

gh api repos/{owner}/{repo}/statuses/COMMITSHA \
    -f state='failure' \
    -f context='External job' \
    -f description='Tests failed'

on failure

or

gh api repos/{owner}/{repo}/statuses/COMMITSHA \
    -f state='success' \
    -f context='External job' \
    -f description='Tests passed'

on success.