Search code examples
pythondockerazure-pipelines

Environment not building on pipeline after not changes that would effect the build process


I have made some minor updates to a python application I'm working on, nothing that requires new dependencies and I have tested this by branching up a previous a commit that was successfully built on the same pipeline and it still failed. So I believe something has changed within docker to cause this and I have no idea what it is.

Little background around my changes was that i changed some logic around how i was grabbing data in pandas and added a filter around some warnings, that was really it, did not require any new packages to complete this task.

This is my docker file which I have redacted the env variables for due to them being very private :

FROM python:3.9.13 AS compile-image

LABEL Name=paddybot Version=0.1
EXPOSE 80

#Run environment updates and installs
RUN apt-get update && \ 
    apt-get install build-essential -y && \
    apt-get install -y --no-install-recommends \
    unixodbc-dev \
    unixodbc \
    libpq-dev 

#Miniconda
ENV CONDA_DIR /opt/conda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh &&\
     /bin/bash ~/miniconda.sh -b -p /opt/conda
ENV PATH=$CONDA_DIR/bin:$PATH


RUN pip install --upgrade pip
RUN conda config --set restore_free_channel true

#Run requirments install
COPY requirements.txt .

RUN pip install -r requirements.txt

ENV client_id {secrets}
ENV client_secret {secrets}
ENV market_id {secrets}
ENV tenant_id {secrets}
ENV vault_url {secrets}

ADD . /app
WORKDIR /app

CMD ["python", "app.py"]

Here is my build YAML:

    trigger:
    - main

    resources:
    - repo: self

    variables:
       # Container registry service connection established during pipeline creation
       dockerRegistryServiceConnection: {secret}
       imageRepository: 'paddybot'
       containerRegistry: {secret}
       dockerfilePath: '$(Build.SourcesDirectory)/Dockerfile'
       tag: 'latest'
       vmImageName: 'ubuntu-latest'

   stages:
   - stage: Build
   displayName: Build Bot
   jobs:
   - job: Build
    displayName: Build
    steps:
    - task: Docker@2
      displayName: Build and push an image to container registry
      inputs:
        dockerfile: '$(Build.SourcesDirectory)/Dockerfile'

Here is my requirements file - Note i recently changed catboost from v1.2.2 to 1.2.3 as i thought maybe catboost could of been the issue at first:

azure-common==1.1.28
azure-identity==1.13.0
azure-keyvault-secrets==4.7.0
azure-mgmt-resource==22.0.0
msrestazure==0.6.4
azure-storage-blob==12.18.3
catboost==1.2.3
docker==6.1.3
hmmlearn==0.3.0
hyperopt==0.2.5
ipython==8.12.3
ipython-genutils==0.2.0
ipywidgets==8.0.4
joblib==1.2.0
json5==0.9.6
jsonpickle==3.0.1
jsonschema==4.17.3
jupyter==1.0.0
jupyter-client==7.1.2
jupyter-console==6.4.3
jupyter_core==5.4.0
jupyter-server==1.13.5
jupyterlab==3.3.2
jupyterlab-pygments==0.1.2
jupyterlab-server==2.10.3
jupyterlab-widgets==3.0.5
jupyterthemes==0.20.0
keras==2.13.1
keras2onnx==1.6.0
lightgbm==4.1.0
matplotlib
notebook==6.5.4
notebook_shim==0.2.2
numpy==1.24.4
oandapyV20==0.7.2
oauthlib==3.2.2
optuna==3.2.0
pandas==1.1.5
pandas-ta==0.3.14b0
pickle4==0.0.1
pickleshare==0.7.5
plotly==5.17.0
ply==3.11
python-dateutil==2.8.2
pytz==2022.7
requests==2.27.1
requests-oauthlib==1.3.1
scikit-learn==1.3.2
scikit-plot==0.3.7
seaborn==0.12.2
sklearn-pandas==1.7.0
statsmodels
ta==0.10.2
tblib==2.0.0
urllib3==1.26.8
xgboost==1.7.5

Here is my the error im seeing after running the pipeline :

##[error]#7 9.035 Installing base environment...
##[error]#7 9.035 
##[error]#7 10.21 
##[error]#7 10.21 Downloading and Extracting Packages: ...working... done
##[error]#7 10.21 
##[error]#7 10.21 Downloading and Extracting Packages: ...working... done
##[error]done
##[error]done
##[error]#7 15.34 installation finished.
##[error]#7 DONE 15.6s
##[error]#8 [4/9] RUN pip install --upgrade pip
##[error]#8 0.776 Requirement already satisfied: pip in /opt/conda/lib/python3.12/site-packages (23.3.1)
##[error]#8 0.922 Collecting pip
##[error]#8 0.991   Downloading pip-24.0-py3-none-any.whl.metadata (3.6 kB)
##[error]#8 1.013 Downloading pip-24.0-py3-none-any.whl (2.1 MB)
##[error]#8 1.365 Installing collected packages: pip
##[error]#8 1.365   Attempting uninstall: pip
##[error]#8 1.383     Found existing installation: pip 23.3.1
##[error]#8 1.386     Uninstalling pip-23.3.1:
##[error]#8 1.416       Successfully uninstalled pip-23.3.1
##[error]#8 3.086 Successfully installed pip-24.0
##[error]#8 3.086 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
##[error]#8 DONE 3.6s
##[error]#9 [5/9] RUN conda config --set restore_free_channel true
##[error]#9 DONE 0.8s
##[error]#10 [6/9] COPY requirements.txt .
##[error]#10 DONE 0.0s
##[error]#11 [7/9] RUN pip install -r requirements.txt
##[error]#11 0.944 Collecting azure-common==1.1.28 (from -r requirements.txt (line 1))
##[error]#11 1.004   Downloading azure_common-1.1.28-py2.py3-none-any.whl.metadata (5.0 kB)
##[error]#11 1.050 Collecting azure-identity==1.13.0 (from -r requirements.txt (line 2))
##[error]#11 1.057   Downloading azure_identity-1.13.0-py3-none-any.whl.metadata (67 kB)
##[error]#11 1.082      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.1/67.1 kB 2.9 MB/s eta 0:00:00
##[error]#11 1.114 Collecting azure-keyvault-secrets==4.7.0 (from -r requirements.txt (line 3))
##[error]#11 1.120   Downloading azure_keyvault_secrets-4.7.0-py3-none-any.whl.metadata (28 kB)
##[error]#11 1.173 Collecting azure-mgmt-resource==22.0.0 (from -r requirements.txt (line 4))
##[error]#11 1.194   Downloading azure_mgmt_resource-22.0.0-py3-none-any.whl.metadata (30 kB)
##[error]#11 1.238 Collecting msrestazure==0.6.4 (from -r requirements.txt (line 5))
##[error]#11 1.245   Downloading msrestazure-0.6.4-py2.py3-none-any.whl.metadata (15 kB)
##[error]#11 1.294 Collecting azure-storage-blob==12.18.3 (from -r requirements.txt (line 6))
##[error]#11 1.300   Downloading azure_storage_blob-12.18.3-py3-none-any.whl.metadata (26 kB)
##[error]#11 1.451 Collecting catboost==1.2.3 (from -r requirements.txt (line 7))
##[error]#11 1.459   Downloading catboost-1.2.3-cp312-cp312-manylinux2014_x86_64.whl.metadata (1.2 kB)
##[error]#11 1.509 Collecting docker==6.1.3 (from -r requirements.txt (line 8))
##[error]#11 1.547   Downloading docker-6.1.3-py3-none-any.whl.metadata (3.5 kB)
##[error]#11 1.587 Collecting hmmlearn==0.3.0 (from -r requirements.txt (line 9))
##[error]#11 1.595   Downloading hmmlearn-0.3.0.tar.gz (77 kB)
##[error]#11 1.606      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 8.4 MB/s eta 0:00:00
##[error]#11 1.630   Preparing metadata (setup.py): started
##[error]#11 4.626   Preparing metadata (setup.py): finished with status 'done'

##[error]#11 4.654 Collecting hyperopt==0.2.5 (from -r requirements.txt (line 10))
##[error]#11 4.750   Downloading hyperopt-0.2.5-py2.py3-none-any.whl.metadata (1.6 kB)
##[error]#11 4.832 Collecting ipython==8.12.3 (from -r requirements.txt (line 11))
##[error]#11 4.838   Downloading ipython-8.12.3-py3-none-any.whl.metadata (5.7 kB)
##[error]#11 4.868 Collecting ipython-genutils==0.2.0 (from -r requirements.txt (line 12))
##[error]#11 4.875   Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
##[error]#11 4.931 Collecting ipywidgets==8.0.4 (from -r requirements.txt (line 13))
##[error]#11 4.938   Downloading ipywidgets-8.0.4-py3-none-any.whl.metadata (2.3 kB)
##[error]#11 4.984 Collecting joblib==1.2.0 (from -r requirements.txt (line 14))
##[error]#11 5.000   Downloading joblib-1.2.0-py3-none-any.whl.metadata (5.3 kB)
##[error]#11 5.035 Collecting json5==0.9.6 (from -r requirements.txt (line 15))
##[error]#11 5.043   Downloading json5-0.9.6-py2.py3-none-any.whl.metadata (8.0 kB)
##[error]#11 5.082 Collecting jsonpickle==3.0.1 (from -r requirements.txt (line 16))
##[error]#11 5.090   Downloading jsonpickle-3.0.1-py2.py3-none-any.whl.metadata (7.5 kB)
##[error]#11 5.146 Collecting jsonschema==4.17.3 (from -r requirements.txt (line 17))
##[error]#11 5.162   Downloading jsonschema-4.17.3-py3-none-any.whl.metadata (7.9 kB)
##[error]#11 5.197 Collecting jupyter==1.0.0 (from -r requirements.txt (line 18))
##[error]#11 5.205   Downloading jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)
##[error]#11 5.267 Collecting jupyter-client==7.1.2 (from -r requirements.txt (line 19))
##[error]#11 5.274   Downloading jupyter_client-7.1.2-py3-none-any.whl.metadata (5.4 kB)
##[error]#11 5.308 Collecting jupyter-console==6.4.3 (from -r requirements.txt (line 20))
##[error]#11 5.317   Downloading jupyter_console-6.4.3-py3-none-any.whl.metadata (1.2 kB)
##[error]#11 5.364 Collecting jupyter_core==5.4.0 (from -r requirements.txt (line 21))
##[error]#11 5.372   Downloading jupyter_core-5.4.0-py3-none-any.whl.metadata (3.4 kB)
##[error]#11 5.458 Collecting jupyter-server==1.13.5 (from -r requirements.txt (line 22))
##[error]#11 5.477   Downloading jupyter_server-1.13.5-py3-none-any.whl.metadata (4.3 kB)
##[error]#11 5.678 Collecting jupyterlab==3.3.2 (from -r requirements.txt (line 23))
##[error]#11 5.687   Downloading jupyterlab-3.3.2-py3-none-any.whl.metadata (11 kB)
##[error]#11 5.716 Collecting jupyterlab-pygments==0.1.2 (from -r requirements.txt (line 24))
##[error]#11 5.727   Downloading jupyterlab_pygments-0.1.2-py2.py3-none-any.whl.metadata (329 bytes)
##[error]#11 5.797 Collecting jupyterlab-server==2.10.3 (from -r requirements.txt (line 25))
##[error]#11 5.805   Downloading jupyterlab_server-2.10.3-py3-none-any.whl.metadata (2.9 kB)
##[error]#11 5.851 Collecting jupyterlab-widgets==3.0.5 (from -r requirements.txt (line 26))
##[error]#11 5.860   Downloading jupyterlab_widgets-3.0.5-py3-none-any.whl.metadata (4.1 kB)
##[error]#11 5.911 Collecting jupyterthemes==0.20.0 (from -r requirements.txt (line 27))
##[error]#11 5.920   Downloading jupyterthemes-0.20.0-py2.py3-none-any.whl (7.0 MB)
##[error]#11 6.196 Collecting keras==2.13.1 (from -r requirements.txt (line 28))
##[error]#11 6.203   Downloading keras-2.13.1-py3-none-any.whl.metadata (2.4 kB)
##[error]#11 6.308 Collecting keras2onnx==1.6.0 (from -r requirements.txt (line 29))
##[error]#11 6.317   Downloading keras2onnx-1.6.0-py3-none-any.whl (219 kB)
##[error]#11 6.327      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 219.8/219.8 kB 32.8 MB/s eta 0:00:00
##[error]#11 6.368 Collecting lightgbm==4.1.0 (from -r requirements.txt (line 30))
##[error]#11 6.376   Downloading lightgbm-4.1.0-py3-none-manylinux_2_28_x86_64.whl.metadata (19 kB)
##[error]#11 6.559 Collecting matplotlib (from -r requirements.txt (line 31))
##[error]#11 6.634 Collecting notebook==6.5.4 (from -r requirements.txt (line 32))
##[error]#11 6.719   Downloading notebook-6.5.4-py3-none-any.whl.metadata (2.5 kB)
##[error]#11 6.751 Collecting notebook_shim==0.2.2 (from -r requirements.txt (line 33))
##[error]#11 6.758   Downloading notebook_shim-0.2.2-py3-none-any.whl.metadata (4.0 kB)
##[error]#11 6.993 Collecting numpy==1.24.4 (from -r requirements.txt (line 34))
##[error]#11 7.006   Downloading numpy-1.24.4.tar.gz (10.9 MB)
##[error]#11 8.246   Installing build dependencies: started
##[error]#11 11.71   Installing build dependencies: finished with status 'done'
##[error]#11 11.72   Getting requirements to build wheel: started
##[error]#11 11.90   Getting requirements to build wheel: finished with status 'error'
##[error]#11 11.90   error: subprocess-exited-with-error
##[error]#11 11.90   
##[error]#11 11.90   × Getting requirements to build wheel did not run successfully.
##[error]#11 11.90   │ exit code: 1
##[error]#11 11.90   ╰─> [33 lines of output]
##[error]#11 11.90       Traceback (most recent call last):
##[error]#11 11.90         File "/opt/conda/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>

##[error]#11 11.90           main()
##[error]#11 11.90         File "/opt/conda/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
##[error]#11 11.90           json_out['return_val'] = hook(**hook_input['kwargs'])
##[error]#11 11.90                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##[error]#11 11.90         File "/opt/conda/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
##[error]#11 11.90           backend = _build_backend()
##[error]#11 11.90                     ^^^^^^^^^^^^^^^^
##[error]#11 11.90         File "/opt/conda/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
##[error]#11 11.90           obj = import_module(mod_path)
##[error]#11 11.90                 ^^^^^^^^^^^^^^^^^^^^^^^
##[error]#11 11.90         File "/opt/conda/lib/python3.12/importlib/__init__.py", line 90, in import_module
##[error]#11 11.90           return _bootstrap._gcd_import(name[level:], package, level)
##[error]#11 11.90                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
##[error]#11 11.90         File "<frozen importlib._bootstrap_external>", line 994, in exec_module
##[error]#11 11.90         File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
##[error]#11 11.90         File "/tmp/pip-build-env-hivx3npt/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
##[error]#11 11.90           import setuptools.version
##[error]#11 11.90         File "/tmp/pip-build-env-hivx3npt/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
##[error]#11 11.90           import pkg_resources
##[error]#11 11.90         File "/tmp/pip-build-env-hivx3npt/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
##[error]#11 11.90           register_finder(pkgutil.ImpImporter, find_on_path)
##[error]#11 11.90                           ^^^^^^^^^^^^^^^^^^^
##[error]#11 11.90       AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
##[error]#11 11.90       [end of output]
##[error]#11 11.90   
##[error]#11 11.90   note: This error originates from a subprocess, and is likely not a problem with pip.
##[error]#11 11.91 error: subprocess-exited-with-error
##[error]#11 11.91 
##[error]#11 11.91 × Getting requirements to build wheel did not run successfully.
##[error]#11 11.91 │ exit code: 1
##[error]#11 11.91 ╰─> See above for output.
##[error]#11 11.91 
##[error]#11 11.91 note: This error originates from a subprocess, and is likely not a problem with pip.
##[error]#11 ERROR: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1
##[error]------
##[error] > [7/9] RUN pip install -r requirements.txt:
##[error]11.90       [end of output]
##[error]11.90   
##[error]11.90   note: This error originates from a subprocess, and is likely not a problem with pip.
##[error]11.91 error: subprocess-exited-with-error
##[error]11.91 
##[error]11.91 × Getting requirements to build wheel did not run successfully.
##[error]11.91 │ exit code: 1
##[error]11.91 ╰─> See above for output.
##[error]11.91 
##[error]11.91 note: This error originates from a subprocess, and is likely not a problem with pip.
##[error]------
##[error]Dockerfile:31
##[error]--------------------
##[error]  29 |     #Run requirments install
##[error]  30 |     COPY requirements.txt .
##[error]  31 | >>> RUN pip install -r requirements.txt
##[error]  32 |     
##[error]  33 |     ENV client_id 334ad5b6-537d-408f-a089-cee496a50532
##[error]--------------------
##[error]ERROR: failed to solve: process "/bin/sh -c pip install -r requirements.txt" did not complete successfully: exit code: 1
##[error]The process '/usr/bin/docker' failed with exit code 1

Solution

  • I have been able to solve this problem thanks to comments.

    Even though I am defining a version for python here

    FROM python:3.9.13 AS compile-image
    

    when installing MiniConda from this line

    RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh &&\
     /bin/bash ~/miniconda.sh -b -p /opt/conda
    

    This automatically downloads python version 3.12/latest and the current version of numpy I am using as shown in the comments can only be used for version 3.8-3.11

    I has change my code to this

    RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-py39_23.11.0-2-Linux-x86_64.sh -O ~/miniconda.sh &&\
    /bin/bash ~/miniconda.sh -b -p /opt/conda
    

    this would then use Py version 3.9 when downloading Miniconda and successfully building