Best practice for creating new release in gitlab CI/CD-pipeline

I have a python-project on gitlab, with an integrated pipeline to both run tests, and if these tests are successful and are run on the main branch, the documentation and a wheel file should be build. Now, however, I would like to expand this by automatically creating releases for new versions and compiling the project into an executable. My current .gitlab-ci.yml-file looks like this:

# This file is a template, and might need editing before it works on your project.
# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Python.gitlab-ci.yml

# Official language image. Look for the different tagged releases at:
# https://hub.docker.com/r/library/python/tags/
image: python:3.11

# Change pip's cache directory to be inside the project directory since we can
# only cache local items.
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

# Pip's cache doesn't store the python packages
# https://pip.pypa.io/en/stable/topics/caching/
#
# If you want to also cache the installed packages, you have to install
# them in a virtualenv and cache it as well.
cache:
  paths:
    - /root/.cache/pypoetry
    - .cache/pip
    - venv/

before_script:
  #- ubuntu-drivers install --gpgpu nvidia:555 -y
  - apt-get update && apt-get install software-properties-common python3-launchpadlib -y
  - apt-get update && add-apt-repository main contrib non-free non-free-firmware # ppa:graphics-drivers/ppa
  - echo "deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware" >> /etc/apt/sources.list
  - cat /etc/apt/sources.list
  #- deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
  - apt-get update && apt-get install ffmpeg libsm6 libxext6 libegl-dev -y #nvidia-driver firmware-misc-nonfree -y -o Dpkg::Options::="--force-overwrite" # nvidia-dkms nvidia-utils -y
  - nvidia-smi # For debugging
  - python --version # For debugging
  - pip install --upgrade pip
  - pip install poetry
  - poetry install --with dev
  - source `poetry env info --path`/bin/activate

stages:
  - test
  - wheel_build
  - doc_build
  - nuitka_windows
  - nuitka_linux
  - release

test_job:
  image: "python:$VERSION"
  stage: test
  script:
    - poetry run pytest ./tests/
  parallel:
    matrix:
      - VERSION: ["3.9", "3.10", "3.11", "3.12"]
  artifacts:
    when: always
    reports:
      junit: /builds/my_python_project/junit_report.xml
      coverage_report:
        coverage_format: cobertura
        path: /builds/my_python_project/coverage.xml
  coverage: '/TOTAL.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'

wheel_build_job:
  stage: wheel_build
  needs: [test_job]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - poetry build
    - pwd
  artifacts:
    paths:
      - /builds/my_python_project/dist/*.whl

doc_build_job:
  stage: doc_build
  needs: [test_job]
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  script:
    - cd docs
    - poetry run sphinx-apidoc -o ./source/ ../my_python_project/
    - poetry run sphinx-build -M html ./source/ ./build/
    - mv build/html/ ../public/
  artifacts:
    paths:
      - /builds/my_python_project/public

nuitka_job_windows:
  stage: nuitka_windows
  tags: [windows]
  needs: [wheel_build_job]
  rules: 
    - if: '($CI_COMMIT_TAG != null) && ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH)'
  script:
    - echo "Hello World from Windows"

nuitka_job_linux:
  stage: nuitka_linux
  tags: [linux]
  needs: [wheel_build_job]
  rules: 
    - if: '($CI_COMMIT_TAG != null) && ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH)'
  script:
    - echo "Hello World from Linux"

release_job:
  stage: release
  needs: 
    - job: wheel_build_job
      optional: false
    - job: doc_build_job
      optional: false
    - job: nuitka_job_linux
      optional: true
    - job: nuitka_job_windows
      optional: true
  rules:
    - if: '($CI_COMMIT_TAG != null) && ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH)'# Run this job when a tag is created
  script:
    - echo "running release_job"
    - echo "Current commit tag is $CI_COMMIT_TAG"
    - curl --location --output /usr/local/bin/release-cli "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages/generic/release-cli/latest/release-cli-linux-amd64"
    - chmod +x /usr/local/bin/release-cli
    - release-cli -v
  release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
    tag_name: "$CI_COMMIT_TAG"
    description: "$CI_COMMIT_DESCRIPTION"

My first idea was to create tags before merging from a feature branch, and thereby trigger all additional stages. However, this tag does not seem to be transferred to the main branch, thereby staying empty and never triggering the additional stages. Thus, my current approach would be to create a new branch explicitly for new releases (i.e. named release/v<x.xx.xx>), and merge to main from there. However, I don't know yet how to extract the relevant information to determine if I'm merging from the correct branch to trigger the additional steps, and to extract the correct version number.

Is that the correct approach at all, or are there other, better implementations for my intentions? I looked for potential tutorials, but did not find anything useful yet.

Solution

I am not sure that I totally got you but I think the approach should be:

a feature branch merge request pipeline runs the build and test jobs.
After merging to a main branch, the pipeline on the main branch should create a tag (using the Gitlab API)
As a result of the tag, a new pipeline will be created running any additional release related tasks like a release in Gitlab for example.

Taking a look at your configuration I think some of the rules are wrong, for example

release_job:
  rules:
    - if: '($CI_COMMIT_TAG != null) && ($CI_MERGE_REQUEST_TARGET_BRANCH_NAME == $CI_DEFAULT_BRANCH)'# Run this job when a tag is created

This job should ran as a result of the new tag, meaning after the MR was merged, meaning there is no $CI_MERGE_REQUEST_TARGET_BRANCH_NAME . You could try to replace it with $CI_COMMIT_BRANCH if you have to make sure it run on the default branch. Of course this is just the general idea and there are many possible considerations to make, like how to handle artifacts/build results, or how to generate the next semantic version, etc.