Good day everyone, hope you're doing really well. I would like to start this question by prefacing that I'm a real newbie when it comes to setting up projects in the right way. Right now I'm working on a Django app with a friend that already has some more experience with web development and he had already set up everything with Docker-compose, that is, we have containers for our Django app, MySQL, Celery, RabbitMQ, etc.
Now, I'm working on the back-end side of our project and need to add a new package to the list of packages the app container has: Smart-Open. The thing is, I have no idea how to do that. I'm on Windows, and using Git Bash to launch the docker-compose containers. What I've tried is opened the bash of the web app container and tried to pipenv install the module from there, but it was extremely slow, which made the Pipfile updating timeout:
$ winpty docker exec -it whs_webapp_1 bash
root@96caa176d252:/app/my_app_name# pipenv install smart_open[s3]
Creating a virtualenv for this project…
Using /usr/local/bin/python3.9 (3.9.6) to create virtualenv…
⠋Running virtualenv with interpreter /usr/local/bin/python3.9
Using base prefix '/usr/local'
/usr/lib/python3/dist-packages/virtualenv.py:1090: DeprecationWarning: the imp m
import imp
New python executable in /root/.local/share/virtualenvs/my_app-NMSbRsZM/bi
Also creating executable in /root/.local/share/virtualenvs/my_app-NMSbRsZM
Installing setuptools, pkg_resources, pip, wheel...done.
Virtualenv location: /root/.local/share/virtualenvs/my_app-NMSbRsZM
Installing smart_open[s3]…
Collecting smart_open[s3]
Using cached smart_open-5.1.0-py3-none-any.whl (57 kB)
Collecting boto3
Downloading boto3-1.18.10-py3-none-any.whl (131 kB)
Collecting botocore<1.22.0,>=1.21.10
Downloading botocore-1.21.10-py3-none-any.whl (7.8 MB)
Collecting s3transfer<0.6.0,>=0.5.0
Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting jmespath<1.0.0,>=0.7.1
Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting six>=1.5
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, urllib3, python-dateutil, jmespath, botocore
Successfully installed boto3-1.18.10 botocore-1.21.10 jmespath-0.10.0 python-dat
Adding smart_open[s3] to Pipfile's [packages]…
Pipfile.lock (acbafc) out of date, updating to (9b92f3)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
items())[0][1]
File "/usr/lib/python3/dist-packages/pipenv/patched/piptools/resolver.py", lin
return {ireq: self.repository.get_hashes(ireq) for ireq in ireqs}
File "/usr/lib/python3/dist-packages/pipenv/patched/piptools/resolver.py", lin
return {ireq: self.repository.get_hashes(ireq) for ireq in ireqs}
File "/usr/lib/python3/dist-packages/pipenv/patched/piptools/repositories/pypi
return {
File "/usr/lib/python3/dist-packages/pipenv/patched/piptools/repositories/pypi
self._get_file_hash(candidate.location)
File "/usr/lib/python3/dist-packages/pipenv/patched/piptools/repositories/pypi
for chunk in iter(lambda: fp.read(8096), b""):
File "/usr/lib/python3/dist-packages/pipenv/patched/piptools/repositories/pypi
for chunk in iter(lambda: fp.read(8096), b""):
File "/usr/lib/python3/dist-packages/pipenv/vendor/pip9/_vendor/requests/packa
flush_decoder = True
File "/usr/local/lib/python3.9/contextlib.py", line 135, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/lib/python3/dist-packages/pipenv/vendor/pip9/_vendor/requests/packa
raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip9._vendor.requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnect
This command was running for about two hours until the timout happened. Can anyone explain to me how I exactly do I add this new package correctly? I can share our docker configuration, but I'm not sure what might be useful. Basically, what's the right way to add an additional package to the pipfile and have pipfile.lock update accordingly?
Thank you in advance for any help you may have. I've searched the internet but most guides do not seem to compare to the config we have, so I'm as confused as I can be.
Does your project directory contain a docker-compose.yml, a Dockerfile, and a requirements.txt file? If so, then the following steps might help.
Open your requirements.txt. You should see all of the python project dependencies listed here. For example, it might look like this initially:
Django==3.1.5
djangorestframework==3.12.2
Add smart-open as a requirement so that your requirements.txt looks something like:
Django==3.1.5
djangorestframework==3.12.2
smart-open
Now in order to update your image with this new package, you would need to rebuild the image. This is typically done as:
docker-compose up --build
Note: the above command will re-create the original image but with smart-open installed and will then spin up a container called whs_webapp_1 using this image.
To confirm that the package has been installed you can bash into the container and then open a python interactive terminal, by running the following command:
docker exec -it whs_webapp_1 bash
python (or python3)
and testing that your package can be imported.