Search code examples
pythondockermysql-python

How to install mysqlclient in a python:3-slim Docker image without bloating the image?

I'm using python:3-slim Docker image and want to use the mysqlclient package from Pypi but getting the following error from RUN pip install mysqlclient command:

...
Collecting mysqlclient
  Downloading mysqlclient-2.2.0.tar.gz (89 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.5/89.5 kB 2.5 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      /bin/sh: 1: pkg-config: not found
      /bin/sh: 1: pkg-config: not found
      Trying pkg-config --exists mysqlclient
      Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
      Trying pkg-config --exists mariadb
      Command 'pkg-config --exists mariadb' returned non-zero exit status 127.
      Traceback (most recent call last):
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-f_fea8lo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-f_fea8lo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-f_fea8lo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 154, in <module>
        File "<string>", line 48, in get_config_posix
        File "<string>", line 27, in find_package_name
      Exception: Can not find valid pkg-config name.
      Specify MYSQLCLIENT_CFLAGS and MYSQLCLIENT_LDFLAGS env vars manually
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

My Dockerfile looks like:

FROM python:3.12-slim

RUN pip install mysqlclient

I tried installing the build dependencies by adding the following to the Dockerfile above RUN pip install mysqlclient:

RUN apt-get install python3-dev default-libmysqlclient-dev build-essential

but this took the Docker image to over 700MB. I also tried replacing the line above with the following to remove the build dependencies after install mysqlclient:

RUN apt-get update && \
    apt-get dist-upgrade && \
    apt-get install -y pkg-config default-libmysqlclient-dev \
        build-essential libmariadb-dev && \
    pip install mysqlclient && \
    apt-get purge -y pkg-config default-libmysqlclient-dev build-essential

but the resulting image was still over 500MB in size and took a lot of precious CI/CD build time as it installs and then uninstalls the build tools.

How do I install mysqlclient without bloating my docker image with build dependencies?

Solution

  • Use a Multi-Stage build Dockerfile:

    FROM python:3.12 AS python-build
RUN pip install mysqlclient

FROM python:3.12-slim
COPY --from=python-build /usr/local/lib/python3.12/site-packages /usr/local/lib/python3.12/site-packages
RUN apt-get update && apt-get install -y libmariadb3

    This first 'stage' uses a full-fat python:3.12 image, which has the necessary build tools to install and compile mysqlclient. As is typical with Python, the mysqlclient package along with its dependencies are installed in /usr/local/lib/python*/site-packages.

    Then the build starts again, this time using python:3.12-slim image. The COPY references the first image and copies just the installed Python packages including mysqlclient. The final line installs libmariadb3, which is required by mysqlclient, otherwise you'll receive the following error:

    ImportError: libmariadb.so.3: cannot open shared object file: No such file or directory

    Your image size should now be a little over 190MB. You'll find it'll build much quicker too.