I'm trying to deploy a django application on heroku. I use docker and poetry. I have a problem because in the dockerfile I install poetry and install dependencies using poetry. Locally it works fine, but when I want to deploy on heroku, the dependencies are not installed. Only putting code like:
'RUN pip install -r requirements.txt' installs the dependencies on heroku (I created the requirements.txt file using poetry export).
Should I have two different Dockerfiles for dev and prod, where I will use poetry in one and pip install requirements in the other ? Or should I use multistaged Dockerfile or do you have any other suggestions ?
My Dockerfile with potery:
FROM python:3.11-alpine
WORKDIR /code
ENV PYTHONUNBUFFERED 1
ENV PATH "/root/.local/bin:$PATH"
RUN apk add --no-cache curl \
&& curl -sSL https://install.python-poetry.org | python3 - \
&& apk add --no-cache postgresql-dev musl-dev
COPY poetry.lock pyproject.toml /code/
RUN poetry install --no-root
COPY . /code/
RUN poetry install
heroku.yml:
build:
docker:
web: backend/Dockerfile
run:
web: gunicorn core.wsgi:application --bind 0.0.0.0:$PORT
Output from deploy:
$ git push heroku main
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 12 threads
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 1.09 KiB | 1.09 MiB/s, done.
Total 6 (delta 3), reused 0 (delta 0), pack-reused 0
remote: Updated 63 paths from 075607f
remote: Compressing source files... done.
remote: Building source:
remote: Waiting on build...
remote: Waiting on build... (elapsed: 6s)
remote: Waiting on build... (elapsed: 9s)
remote: Waiting on build... (elapsed: 12s)
remote: === Fetching app code
remote:
remote: === Building web (backend/Dockerfile)
remote:
remote: Sending build context to Docker daemon 53.76kB
remote: Step 1/10 : FROM python:3.11-alpine
remote: 3.11-alpine: Pulling from library/python
remote: 4abcf2066143: Pulling fs layer
remote: c3cdf40b8bda: Pulling fs layer
remote: ac499ccf2147: Pulling fs layer
remote: 416bfceb623e: Pulling fs layer
remote: 76351c33299b: Pulling fs layer
remote: 416bfceb623e: Waiting
remote: 76351c33299b: Waiting
remote: c3cdf40b8bda: Download complete
remote: 4abcf2066143: Verifying Checksum
remote: 4abcf2066143: Download complete
remote: 416bfceb623e: Download complete
remote: ac499ccf2147: Verifying Checksum
remote: ac499ccf2147: Download complete
remote: 76351c33299b: Verifying Checksum
remote: 76351c33299b: Download complete
remote: 4abcf2066143: Pull complete
remote: c3cdf40b8bda: Pull complete
remote: ac499ccf2147: Pull complete
remote: 416bfceb623e: Pull complete
remote: 76351c33299b: Pull complete
remote: Digest: sha256:0b5ed25d3cc27cd35c7b0352bac8ef2ebc8dd3da72a0c03caaf4eb15d9ec827a
remote: Status: Downloaded newer image for python:3.11-alpine
remote: ---> 10333afc009e
remote: Step 2/10 : WORKDIR /code
remote: ---> Running in ed5fbef22730
remote: Removing intermediate container ed5fbef22730
remote: ---> e217b7efac9b
remote: Step 3/10 : ENV DJANGO_SETTINGS_MODULE core.settings
remote: ---> Running in 35d73be810f5
remote: Removing intermediate container 35d73be810f5
remote: ---> 06f7f12a8f60
remote: Step 4/10 : ENV PYTHONUNBUFFERED 1
remote: ---> Running in 13220906f1c2
remote: Removing intermediate container 13220906f1c2
remote: ---> c7fd5fadf540
remote: Step 5/10 : ENV PATH "/root/.local/bin:$PATH"
remote: ---> Running in 747dae6a957b
remote: Removing intermediate container 747dae6a957b
remote: ---> da2b7e7bfd6a
remote: Step 6/10 : RUN apk add --no-cache curl && curl -sSL https://install.python-poetry.org | python3 - && apk add --no-cache postgresql-dev musl-dev
remote: ---> Running in f57663dadafc
remote: fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
remote: fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
remote: (1/7) Installing brotli-libs (1.1.0-r1)
remote: (2/7) Installing c-ares (1.27.0-r0)
remote: (3/7) Installing libunistring (1.1-r2)
remote: (4/7) Installing libidn2 (2.3.4-r4)
remote: (5/7) Installing nghttp2-libs (1.58.0-r0)
remote: (6/7) Installing libcurl (8.5.0-r0)
remote: (7/7) Installing curl (8.5.0-r0)
remote: Executing busybox-1.36.1-r15.trigger
remote: OK: 20 MiB in 45 packages
remote: Retrieving Poetry metadata
remote:
remote: # Welcome to Poetry!
remote:
remote: This will download and install the latest version of Poetry,
remote: a dependency and package manager for Python.
remote:
remote: It will add the `poetry` command to Poetry s bin directory, located at:
remote:
remote: /root/.local/bin
remote:
remote: You can uninstall at any time by executing this script with the --uninstall option,
remote: and these changes will be reverted.
remote:
remote: Installing Poetry (1.8.2)
remote: Installing Poetry (1.8.2): Creating environment
remote: Installing Poetry (1.8.2): Installing Poetry
remote: Installing Poetry (1.8.2): Creating script
remote: Installing Poetry (1.8.2): Done
remote:
remote: Poetry (1.8.2) is installed now. Great!
remote:
remote: You can test that everything is set up by executing:
remote:
remote: `poetry --version`
remote:
remote: fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/x86_64/APKINDEX.tar.gz
remote: fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/x86_64/APKINDEX.tar.gz
remote: (1/39) Upgrading libcrypto3 (3.1.4-r5 -> 3.1.4-r6)
remote: (2/39) Upgrading libssl3 (3.1.4-r5 -> 3.1.4-r6)
remote: (3/39) Installing musl-dev (1.2.4_git20230717-r4)
remote: (4/39) Installing libpq (16.2-r1)
remote: (5/39) Installing pkgconf (2.1.0-r0)
remote: (6/39) Installing openssl-dev (3.1.4-r6)
remote: (7/39) Installing libpq-dev (16.2-r1)
remote: (8/39) Installing libecpg (16.2-r1)
remote: (9/39) Installing libecpg-dev (16.2-r1)
remote: (10/39) Installing fortify-headers (1.1-r3)
remote: (11/39) Installing clang15-headers (15.0.7-r18)
remote: (12/39) Installing libgcc (13.2.1_git20231014-r0)
remote: (13/39) Installing libstdc++ (13.2.1_git20231014-r0)
remote: (14/39) Installing libxml2 (2.11.7-r0)
remote: (15/39) Installing zstd-libs (1.5.5-r8)
remote: (16/39) Installing llvm15-libs (15.0.7-r12)
remote: (17/39) Installing clang15-libs (15.0.7-r18)
remote: (18/39) Installing jansson (2.14-r4)
remote: (19/39) Installing binutils (2.41-r0)
remote: (20/39) Installing libgomp (13.2.1_git20231014-r0)
remote: (21/39) Installing libatomic (13.2.1_git20231014-r0)
remote: (22/39) Installing gmp (6.3.0-r0)
remote: (23/39) Installing isl26 (0.26-r1)
remote: (24/39) Installing mpfr4 (4.2.1-r0)
remote: (25/39) Installing mpc1 (1.3.1-r1)
remote: (26/39) Installing gcc (13.2.1_git20231014-r0)
remote: (27/39) Installing libstdc++-dev (13.2.1_git20231014-r0)
remote: (28/39) Installing clang15-libclang (15.0.7-r18)
remote: (29/39) Installing clang15 (15.0.7-r18)
remote: (30/39) Installing icu-data-en (74.1-r0)
remote: Executing icu-data-en-74.1-r0.post-install
remote: *
remote: * If you need ICU with non-English locales and legacy charset support, install
remote: * package icu-data-full.
remote: *
remote: (31/39) Installing icu-libs (74.1-r0)
remote: (32/39) Installing icu (74.1-r0)
remote: (33/39) Installing icu-dev (74.1-r0)
remote: (34/39) Installing llvm15 (15.0.7-r12)
remote: (35/39) Installing lz4-libs (1.9.4-r5)
remote: (36/39) Installing lz4-dev (1.9.4-r5)
remote: (37/39) Installing zstd (1.5.5-r8)
remote: (38/39) Installing zstd-dev (1.5.5-r8)
remote: (39/39) Installing postgresql16-dev (16.2-r1)
remote: Executing busybox-1.36.1-r15.trigger
remote: Executing ca-certificates-20230506-r0.trigger
remote: OK: 534 MiB in 82 packages
remote: Removing intermediate container f57663dadafc
remote: ---> 0e39b87928ff
remote: Step 7/10 : COPY poetry.lock pyproject.toml /code/
remote: ---> 2d0fae435427
remote: Step 8/10 : RUN poetry install --no-root
remote: ---> Running in 5ba8992a62d2
remote: Creating virtualenv backend-MATOk_fk-py3.11 in /root/.cache/pypoetry/virtualenvs
remote: Installing dependencies from lock file
remote:
remote: Package operations: 25 installs, 0 updates, 0 removals
remote:
remote: - Installing asgiref (3.8.1)
remote: - Installing sqlparse (0.5.0)
remote: - Installing django (5.0.4)
remote: - Installing iniconfig (2.0.0)
remote: - Installing packaging (24.0)
remote: - Installing pluggy (1.4.0)
remote: - Installing djangorestframework (3.15.1)
remote: - Installing inflection (0.5.1)
remote: - Installing mccabe (0.7.0)
remote: - Installing mypy-extensions (1.0.0)
remote: - Installing pycodestyle (2.11.1)
remote: - Installing pyflakes (3.2.0)
remote: - Installing pytest (8.1.1)
remote: - Installing pytz (2024.1)
remote: - Installing pyyaml (6.0.1)
remote: - Installing typing-extensions (4.11.0)
remote: - Installing uritemplate (4.1.1)
remote: - Installing dj-database-url (2.1.0)
remote: - Installing drf-yasg (1.21.7)
remote: - Installing flake8 (7.0.0)
remote: - Installing gunicorn (22.0.0)
remote: - Installing mypy (1.9.0)
remote: - Installing psycopg2 (2.9.9)
remote: - Installing pytest-django (4.8.0)
remote: - Installing python-dotenv (1.0.1)
remote: Removing intermediate container 5ba8992a62d2
remote: ---> 2c51709ee889
remote: Step 9/10 : COPY . /code/
remote: ---> a3af2a6c5b5c
remote: Step 10/10 : RUN poetry install
remote: ---> Running in 3ecd2f328645
remote: Installing dependencies from lock file
remote:
remote: No dependencies to install or update
remote:
remote: Installing the current project: backend (0.1.0)
remote:
remote: Warning: The current project could not be installed: [Errno 2] No such file or directory: '/code/README.md'
remote: If you do not want to install the current project use --no-root.
remote: If you want to use Poetry only for dependency management but not for packaging, you can disable package mode by setting package-mode = false in your pyproject.toml file.
remote: In a future version of Poetry this warning will become an error!
remote: Removing intermediate container 3ecd2f328645
remote: ---> 542545ea1195
remote: Successfully built 542545ea1195
remote: Successfully tagged a31f1f813825dc98a956f33b6a3822912f137c40:latest
remote:
remote: === Pushing web (backend/Dockerfile)
remote: Tagged image "a31f1f813825dc98a956f33b6a3822912f137c40" as "registry.heroku.com/wanderer-test/web"
remote: Using default tag: latest
remote: The push refers to repository [registry.heroku.com/wanderer-test/web]
remote: 1e4eca4bbbe5: Preparing
remote: 115e0e85b9ba: Preparing
remote: 81bae2c8c9e6: Preparing
remote: bab36042a757: Preparing
remote: 08bf6ffaa8ac: Preparing
remote: 4a770d99ce52: Preparing
remote: dbf59eaac1dd: Preparing
remote: ed829bc3e4b2: Preparing
remote: 4c9c2b9681ab: Preparing
remote: d4fc045c9e3a: Preparing
remote: 4a770d99ce52: Waiting
remote: dbf59eaac1dd: Waiting
remote: ed829bc3e4b2: Waiting
remote: 4c9c2b9681ab: Waiting
remote: d4fc045c9e3a: Waiting
remote: 08bf6ffaa8ac: Pushed
remote: 81bae2c8c9e6: Pushed
remote: 1e4eca4bbbe5: Pushed
remote: 4a770d99ce52: Layer already exists
remote: dbf59eaac1dd: Layer already exists
remote: ed829bc3e4b2: Layer already exists
remote: d4fc045c9e3a: Layer already exists
remote: 4c9c2b9681ab: Layer already exists
remote: 115e0e85b9ba: Pushed
remote: bab36042a757: Pushed
remote: latest: digest: sha256:b9da90ba3205e3c005fee403be4f0e631cd41e1de27ef7c93a2a57d4c4d860c3 size: 2417
remote:
remote: Verifying deploy... done.
I don't have a Heroku dyno to test with at the moment, but I believe you're getting bitten by this:
We strongly recommend testing images locally as a non-root user, as containers are not run with root privileges in Heroku.
(The documentation goes on to explain how to do that but I'm not going to copy it here as it's a bit of a sidebar.)
Since Poetry creates a virtual environment by default, that's where your dependencies are going. The system python, pip, etc. have no idea that they exist.
I suggest getting Poetry to install packages in the system Python environment by disabling its virtualenvs.create setting (emphasis added):
If set to
false, Poetry will not create a new virtual environment. If it detects an already enabled virtual environment or an existing one in{cache-dir}/virtualenvsor{project-dir}/.venvit will install dependencies into them, otherwise it will install dependencies into the systems python environment.
One way to do that is to add a new RUN command, or chain it to the existing one where you run poetry install:
RUN poetry config virtualenvs.create false && poetry install
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Side note: I don't think you need to run poetry install twice. You should be able to get rid of the one with --no-root before your COPY command.)
You also don't need to generate a requirements.txt. Having one alongside your Poetry files is confusing, and it's just one more thing to maintain. I suggest you remove it entirely.