How to maintain glibc and libmusl Python wheels in the same pip repository?

Previously we've used our internal pip repository for source distributions only. Moving forward we want to host wheels as well to accomplish two things:

serve our own code to both (local) developer machines and Alpine Docker environments
create wheels for packages that don't have Alpine wheels

Unfortunately the wheels built with different libraries share the same artifact name and the second one gets rejected by the pip repository:

docker-compose.yml

version: '3'

services:
  build-alpine:
    build: alpine
    image: build-alpine-wheels
    volumes:
      - $PWD/cython:/build
    working_dir: /build
    command: sh -c 'python setup.py bdist_wheel && twine upload --repository-url http://pypi:8080 -u admin -p admin dist/*'

  build-debian:
    build: debian
    image: build-debian-wheels
    volumes:
      - $PWD/cython-debian:/build
    working_dir: /build
    command: bash -c 'sleep 10s && python setup.py bdist_wheel && twine upload --repository-url http://pypi:8080 -u admin -p admin dist/*'

  pypi:
    image: stevearc/pypicloud:1.0.2
    volumes:
      - $PWD/pypi:/etc/pypicloud/

  alpine-test:
    image: build-alpine-wheels
    depends_on:
      - build-alpine
    command: sh -c 'while ping -c1 build-alpine &>/dev/null;  do sleep 1; done; echo "build container finished" && pip install -i http://pypi:8080/pypi --trusted-host pypi cython && cython --version'

  debian-test:
    image: python:3.6
    depends_on:
      - build-debian
    command: bash -c 'while ping -c1 build-debian &>/dev/null;  do sleep 1; done; echo "build container finished" && pip install -i http://pypi:8080/pypi --trusted-host pypi cython && cython --version'

alpine/Dockerfile

FROM python:3.6-alpine

RUN apk add --update --no-cache build-base
RUN pip install --upgrade pip
RUN pip install twine

debian/Dockerfile

FROM python:3.6-slim

RUN apt-get update && apt-get install -y \
    build-essential \
 && rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip
RUN pip install twine

pypi/config.ini

[app:main]
use = egg:pypicloud

pyramid.reload_templates = False
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en

pypi.default_read =
    everyone
pypi.default_write =
    everyone

pypi.storage = file
storage.dir = %(here)s/packages

db.url = sqlite:///%(here)s/db.sqlite

auth.admins =
  admin

user.admin = $6$rounds=535000$sFuRqMc5PbRccW1J$OBCsn8szlBwr4yPP243JPqomapgInRCUavv/p/UErt7I5FG4O6IGSHkH6H7ZPlrMXO1I8p5LYCQQxthgWZtxe1

# For beaker
session.encrypt_key = s0ETvuGG9Z8c6lK23Asxse4QyuVCsI2/NvGiNvvYl8E=
session.validate_key = fJvHQieaa0g3XsdgMF5ypE4pUf2tPpkbjueLQAAHN/k=
session.secure = False
session.invalidate_corrupt = true

###
# wsgi server configuration
###

[uwsgi]
paste = config:%p
paste-logger = %p
master = true
processes = 20
reload-mercy = 15
worker-reload-mercy = 15
max-requests = 1000
enable-threads = true
http = 0.0.0.0:8080
virtualenv = /env

###
# logging configuration
# http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
###

[loggers]
keys = root, botocore, pypicloud

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[logger_pypicloud]
level = DEBUG
qualname = pypicloud
handlers =

[logger_botocore]
level = WARN
qualname = botocore
handlers =

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(levelname)s %(asctime)s [%(name)s] %(message)s

Setup and execution

git clone https://github.com/cython/cython
git clone https://github.com/cython/cython cython-debian
docker-compose build
docker-compose up

At the end I would like both test containers to be able to execute cython --version. Which works for the Alpine container:

alpine-test_1   | Collecting cython
alpine-test_1   | Downloading http://pypi:8080/api/package/cython/Cython-0.29.12-cp36-cp36m-linux_x86_64.whl (5.0MB)
alpine-test_1   | Installing collected packages: cython
alpine-test_1   | Successfully installed cython-0.29.12
alpine-test_1   | Cython version 0.29.12

But doesn't work for the Debian container:

debian-test_1   | Downloading http://pypi:8080/api/package/cython/Cython-0.29.12-cp36-cp36m-linux_x86_64.whl (5.0MB)
debian-test_1   | Installing collected packages: cython
debian-test_1   | Successfully installed cython-0.29.12
debian-test_1   | Traceback (most recent call last):
debian-test_1   |   File "/usr/local/bin/cython", line 6, in <module>
debian-test_1   |     from Cython.Compiler.Main import setuptools_main
debian-test_1   |   File "/usr/local/lib/python3.6/site-packages/Cython/Compiler/Main.py", line 28, in <module>
debian-test_1   |     from .Scanning import PyrexScanner, FileSourceDescriptor
debian-test_1   | ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

I find it particularly curious that both environments try to pull this wheel because there are all sorts of packages which don't work with Alpine (e.g. Pandas) in which case pip goes straight for the source distribution. I suppose I must be doing something wrong in that regard as well.

So now I'm wondering how I can create these wheels such that for each version of the software package two different wheels can live in the pip repository and have pip automatically download and install the correct one.

Solution

There is currently no support for musl in the manylinux standard: your options are to always build from source, or target a different, glibc-based platform.