Search code examples
pythongithubgithub-actionsreleasepypi

GitHub action - publishing to both TestPyPI and PyPI fails

I set up a GitHub workflow for publishing my Python project to (Test)PyPI according to this guide: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ resulting in this workflow: https://github.com/felixocker/ontor/blob/main/.github/workflows/publish-to-pypi.yml

Whenever I push to the main branch, a new TestPyPI release is created:

- name: Publish distribution to Test PyPI
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.TEST_PYPI_API_TOKEN }}
    repository_url: https://test.pypi.org/legacy/

This works just fine, cp., e.g., commit 15a3bf6

When I push with a tag, this should push to TestPyPI as usual, but the tag should also trigger pushing to PyPI:

- name: Publish distribution to PyPI
  if: startsWith(github.ref, 'refs/tags')
  uses: pypa/gh-action-pypi-publish@master
  with:
    password: ${{ secrets.PYPI_API_TOKEN }}

However, this fails with a 400 error, cp., e.g., commit b450e0f:

ERROR    HTTPError: 400 Bad Request from https://test.pypi.org/legacy/
         File already exists. See https://test.pypi.org/help/#file-name-reuse
         for more information.

Interestingly, the error already occurs in the step for pushing to TestPyPI, and pushing to PyPI is skipped after that. Also, when I tried uploading the latest version to PyPI manually, I got the same error. After that manual attempt, the version was available via PyPI, though.

Really not sure what I am missing here - so pointers are greatly appreciated :)

#################################################

Edit #1:

The behavior makes sense to me for the commits 15a3bf6 and b5cade1, as I did not change the version number here.

What I just realized thanks to @Indra's response and the action overview is that the workflow is executed twice for the same commit: b450e0f in runs #33 and #34
Are there two runs because tags are treated as pushes of their own (which might be indicated by "v0.4.2" vs "main")?
I pushed the commit and the respective tag using

git push --atomic origin main <tag>

which should be simultaneous, though.
If so, is there a way to keep the intended behavior: "publish to TestPyPI whenever I push to main, publish to PyPI whenever there is a tag" without having two runs, assuming that I update the version?

Solution

  • So here is what happened:
    Apparently, GitHub treats the pushes for a commit and a tag separately even when using

    git push --atomic origin main <tag>

    via SSH, see this discussion.
    This results in the github action being run twice, as it is triggered by every push. The version number is already being used for TestPyPI in the first run, yielding a file already exists error in the second.

    What worked for me:
    Making the two cases for pushing to TestPyPI and PyPI exclusive according to this discussion as such:

        - name: Publish distribution to Test PyPI
      if: ${{ !startsWith(github.ref, 'refs/tags') }}
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
        repository_url: https://test.pypi.org/legacy/
    - name: Publish distribution to PyPI
      if: startsWith(github.ref, 'refs/tags')
      uses: pypa/gh-action-pypi-publish@master
      with:
        password: ${{ secrets.PYPI_API_TOKEN }}

    Whenever a commit without a tag is pushed to GitHub, this triggers the push to TestPyPI. When a tag is pushed, the version is released to PyPI. A caveat is that the commit must be handled by the action before the tag, which, in my case, did not work for commit f732bda. Also, as mentioned by @Indra, the version number must be incremented for the action to work. Alternatively, a distinction between regular and hierarchical tags, e.g., "v0.3.0/beta", could be made as described here.