How to publish pip wheels using github actions

Question

Maybe a bit of a broad question, but I think this is relevant for any maintainer of python packages that uses github and could reduce their workload significantly, so hopefully the powers that be will let it stand.

Essentially, it seems to me that :

• Building & publishing wheels for python is nice, but sometimes they have to be specific to the operating system for them to work properly (i.e. built on a similar OS/Env)
• Github actions provides a wide range of OSes to run jobs on (for now, for free), so it could be used to build these wheels
• Github actions can be easily used to publish sidst via twine

So my question is:

• Is there a straight-forward way to build wheels for a bunch of environments (e.g. using the OS/env matrix in github actions) then publish them all via github actionse.g. by aggregating them in one place and then running some twine command? The issue seems to be that there's no way to communicated between the different envs/vms setup by github action.

I can think of various solutions involving e.g. an intermeidary s3 bucket, but I'm likely hugely mistaken in regard to how pypi and/or github actions work in this regard so there might be a very simple issue I'm glancing over.

Answer 1

As mentioned in my comment, here is one possible way to run parallel builds but a single upload:

name: 'Aggregation'

on: [push]

env:
  ARTIFACT: artifact.bin

jobs:
  build:
    runs-on: ${{ matrix.os }} 
    strategy:
      matrix:
        os:
          - windows-latest
          - ubuntu-latest
          - macos-latest
    steps:
      - uses: actions/checkout@v2

      - shell: bash
        run: |
          echo "Run your build command here"
          echo "This is a fake ${{ matrix.os }} build artifact" >$ARTIFACT

      - uses: actions/upload-artifact@v2
        with:
          name: build-${{ matrix.os }}-${{ github.sha }}
          path: ${{ env.ARTIFACT }}

  publish:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - uses: actions/download-artifact@v2
        with:
          path: artifacts

      - shell: bash
        working-directory: artifacts
        run: |
          for i in $( ls ); do
            cat $i/$ARTIFACT
          done

Each matrix job build and upload its own artifact to github. The publish job waits for all the previous jobs to complete, before downloading all the artifacts and in this case iterating over them. One added benefit is that if any matrix job fails the publishing will fail. Of course this is simple only if the build steps and commands are the same on all the OSes.

Here are some screenshots of the test run:

The resulting artifacts:

The output of the publish job: