Search code examples
condaminiconda

How do I inform conda to install a later version of apache-beam?

I am a Conda newbie and am trying to familiarise myself with it by using miniconda to install python package apache-beam. I can see at https://anaconda.org/conda-forge/apache-beam that the latest available version is v2.22.0

enter image description here

however when I attempt to install using conda install -c conda-forge/label/cf201901 apache-beam it attempts to install v2.16.0:

# conda install -c conda-forge/label/cf201901 apache-beam
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:

  - defaults/linux-64::mock==2.0.0=py37_0
  - defaults/linux-64::pyarrow==0.13.0=py37he6710b0_0
  - defaults/linux-64::mkl_random==1.1.1=py37h0573a6f_0
  - defaults/linux-64::arrow-cpp==0.13.0=py37h117bdfb_0
  - defaults/noarch::requests==2.25.1=pyhd3eb1b0_0
  - defaults/linux-64::cryptography==2.3.1=py37hc365091_0
  - defaults/noarch::packaging==20.8=pyhd3eb1b0_0
  - defaults/noarch::pytest-xdist==2.2.0=pyhd3eb1b0_0
  - conda-forge/linux-64::behave==1.2.6=py37h89c1867_1003
  - conda-forge/linux-64::parse_type==0.5.2=py37h89c1867_3
  - defaults/linux-64::pytest==6.2.1=py37h06a4308_0
  - defaults/noarch::python-dateutil==2.8.1=py_0
  - defaults/noarch::pytest-forked==1.3.0=py_0
  - defaults/linux-64::mkl-service==2.3.0=py37he8ac12f_0
  - defaults/linux-64::numpy-base==1.19.2=py37hfa32c7d_0
  - defaults/linux-64::pandas==1.1.5=py37ha9443f7_0
  - defaults/linux-64::requests-kerberos==0.12.0=py37_0
  - defaults/noarch::urllib3==1.26.2=pyhd3eb1b0_0
  - defaults/linux-64::numpy==1.19.2=py37h54aff64_0
  - defaults/linux-64::mkl_fft==1.2.0=py37h23d657b_0
  - defaults/noarch::python-hdfs==2.5.8=py_0
  - defaults/linux-64::pyopenssl==19.0.0=py37_0
done

## Package Plan ##

  environment location: /opt/conda/envs/python3

  added / updated specs:
    - apache-beam


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    apache-beam-2.16.0         |   py37h516909a_0         3.4 MB  conda-forge
    grpcio-1.16.0              |   py37hd60e7a3_0         1.0 MB  conda-forge/label/cf201901
    oauth2client-4.1.3         |             py_0          66 KB  conda-forge
    protobuf-3.6.0             |   py37hf484d3e_0         609 KB  main
    six-1.11.0                 |        py37_1001          21 KB  conda-forge/label/cf201901
    ------------------------------------------------------------
                                           Total:         5.1 MB

The following NEW packages will be INSTALLED:

  apache-beam        conda-forge/linux-64::apache-beam-2.16.0-py37h516909a_0
  grpcio             conda-forge/label/cf201901/linux-64::grpcio-1.16.0-py37hd60e7a3_0
  oauth2client       conda-forge/noarch::oauth2client-4.1.3-py_0
  protobuf           main/linux-64::protobuf-3.6.0-py37hf484d3e_0
  six                conda-forge/label/cf201901/linux-64::six-1.11.0-py37_1001

Why is this?

Solution

  • The answer from @FlyingTeller accurately described the problem and a resolution but no harm in sharing a bit of extra info.

    I neglected to say that I'm building my conda env in a docker image and that docker image gets rebuilt in a CI pipeline. In order to ensure I have a repeatable build I have an environment.yaml file in my repo and it is THAT which I use to build the conda env.

    I have had to make changes to that environment.yaml file at various times over the past month (I have installed sh, apache-beam, pytest, pytest-xdict, ipython, behave...and I think that is the order in which I installed them) and my strategy for doing so is to:

    1. docker run myimage:most-recently-built-tag
    2. conda install the required packages
    3. export a new version of environment.yaml
    4. commit new version environment.yaml to the repo

    In doing so I got myself into a position where the version of apache-beam that got installed was 2.16.0 and conflicting dependencies meant I wasn't able to upgrade apache-beam.

    I created a new environment and issued conda install sh apache-beam pytest pytest-xdict ipython behave which successfully resolved all dependencies and installed apache-beam=2.2.0.

    Hence I think my strategy, when I want to make changes to environment.yaml, should be:

    1. create a new conda env
    2. conda install all the required packages in a single command
    3. export a new environment.yaml and commit it to the repo so it can be used to build my docker image in the CI pipeline

    Comments on anything above would be welcomed.

    UPDATE. Here's my Dockerfile that builds a conda environment from which I can export environment.yaml

    FROM debian:buster-slim

RUN apt-get update && apt-get install curl gnupg -y

RUN curl https://repo.anaconda.com/pkgs/misc/gpgkeys/anaconda.asc | gpg --dearmor > conda.gpg && \
        install -o root -g root -m 644 conda.gpg /usr/share/keyrings/conda-archive-keyring.gpg && \
        gpg --keyring /usr/share/keyrings/conda-archive-keyring.gpg --no-default-keyring \
        --fingerprint 34161F5BF5EB1D4BFBBB8F0A8AEB4F8B29D82806 && \
        echo "deb [arch=amd64 signed-by=/usr/share/keyrings/conda-archive-keyring.gpg] https://repo.anaconda.com/pkgs/misc/debrepo/conda stable main" \
        > /etc/apt/sources.list.d/conda.list

RUN MINICONDA_VERSION=4.9.2 && \
        CONDA_VERSION='4.9.*' && \
        CONDA_DIR=/opt/conda && \
        cd /tmp && \
        curl -O https://repo.anaconda.com/miniconda/Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh && \
        /bin/bash Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh -f -b -p $CONDA_DIR && \
        rm Miniconda3-py37_${MINICONDA_VERSION}-Linux-x86_64.sh && \
        $CONDA_DIR/bin/conda config --system --set auto_update_conda false && \
        $CONDA_DIR/bin/conda config --system --set show_channel_urls true && \
        $CONDA_DIR/bin/conda config --system --remove channels defaults && \
        $CONDA_DIR/bin/conda config --system --add channels main && \
        $CONDA_DIR/bin/conda config --system --set env_prompt '({name}) ' && \
        $CONDA_DIR/bin/conda config --system --append envs_dirs /opt/conda/envs/ && \
        $CONDA_DIR/bin/conda config --system --append pkgs_dirs /opt/conda/pkgs/ && \
        $CONDA_DIR/bin/conda update --quiet --yes --all conda="${CONDA_VERSION}" && \
        $CONDA_DIR/bin/conda config --system --append channels conda-forge  && \
        $CONDA_DIR/bin/conda create -n py3 python=3.7 

RUN bash -c "source /opt/conda/bin/activate /opt/conda/envs/py3 && conda install apache-beam sh pytest pytest-xdist ipython behave"

RUN echo "source /opt/conda/bin/activate /opt/conda/envs/py3" >> /root/.bashrc

    and generate my environment.yaml by running:

    docker build . -t conda-beam && \
  docker run --entrypoint bash conda-beam:latest -c \
  "source /opt/conda/bin/activate /opt/conda/envs/py3 \
  && conda env export"

    which, when I ran it just now, returned

    name: py3
channels:
  - main
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - apache-beam=2.22.0=py37h8f50634_0
  - apipkg=1.5=py37_0
  - arrow-cpp=0.15.1=py37h7cd5009_5
  - attrs=20.3.0=pyhd3eb1b0_0
  - backcall=0.2.0=py_0
  - behave=1.2.6=py37h89c1867_1003
  - blas=1.0=mkl
  - boost-cpp=1.71.0=h7b6447c_0
  - brotli=1.0.9=he6710b0_2
  - brotlipy=0.7.0=py37h27cfd23_1003
  - bzip2=1.0.8=h7b6447c_0
  - c-ares=1.17.1=h27cfd23_0
  - ca-certificates=2020.12.8=h06a4308_0
  - certifi=2020.12.5=py37h06a4308_0
  - cffi=1.14.4=py37h261ae71_0
  - chardet=4.0.0=py37h06a4308_1003
  - crcmod=1.7=py37hc8dfbb8_1003
  - cryptography=3.3.1=py37h3c74f83_0
  - cython=0.29.21=py37h2531618_0
  - decorator=4.4.2=py_0
  - dill=0.3.1.1=py37_1
  - docopt=0.6.2=py37_0
  - double-conversion=3.1.5=he6710b0_1
  - execnet=1.7.1=py_0
  - fastavro=0.23.5=py37h7b6447c_0
  - future=0.18.2=py37_1
  - gflags=2.2.2=he6710b0_0
  - glog=0.4.0=he6710b0_0
  - grpc-cpp=1.26.0=hf8bcb03_0
  - grpcio=1.31.0=py37hf8bcb03_0
  - httplib2=0.17.0=py37hc8dfbb8_1
  - icu=58.2=he6710b0_3
  - idna=2.10=py_0
  - importlib-metadata=2.0.0=py_1
  - importlib_metadata=2.0.0=1
  - iniconfig=1.1.1=py_0
  - intel-openmp=2020.2=254
  - ipython=7.19.0=py37hb070fc8_0
  - ipython_genutils=0.2.0=pyhd3eb1b0_1
  - jedi=0.18.0=py37h06a4308_0
  - krb5=1.18.2=h173b8e3_0
  - ld_impl_linux-64=2.33.1=h53a641e_7
  - libboost=1.71.0=h97c9712_0
  - libedit=3.1.20191231=h14c3975_1
  - libevent=2.1.8=h1ba5d50_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.1.0=hdf63c60_0
  - libprotobuf=3.11.2=hd408876_0
  - libstdcxx-ng=9.1.0=hdf63c60_0
  - lz4-c=1.8.1.2=h14c3975_0
  - mkl=2020.2=256
  - mkl-service=2.3.0=py37he8ac12f_0
  - mkl_fft=1.2.0=py37h23d657b_0
  - mkl_random=1.1.1=py37h0573a6f_0
  - mock=2.0.0=py37_0
  - more-itertools=8.6.0=pyhd3eb1b0_0
  - ncurses=6.2=he6710b0_1
  - numpy=1.19.2=py37h54aff64_0
  - numpy-base=1.19.2=py37hfa32c7d_0
  - oauth2client=4.1.3=py_0
  - openssl=1.1.1i=h27cfd23_0
  - packaging=20.8=pyhd3eb1b0_0
  - pandas=1.2.0=py37ha9443f7_0
  - parse=1.18.0=pyh9f0ad1d_0
  - parse_type=0.5.2=py37h89c1867_3
  - parso=0.7.0=py_0
  - pbr=5.5.1=py_0
  - pexpect=4.8.0=pyhd3eb1b0_3
  - pickleshare=0.7.5=pyhd3eb1b0_1003
  - pip=20.3.3=py37h06a4308_0
  - pluggy=0.13.1=py37_0
  - prompt-toolkit=3.0.8=py_0
  - protobuf=3.11.2=py37he6710b0_0
  - ptyprocess=0.7.0=pyhd3eb1b0_2
  - py=1.10.0=pyhd3eb1b0_0
  - pyarrow=0.15.1=py37h0573a6f_0
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.8=py_0
  - pycparser=2.20=py_2
  - pydot=1.3.0=py37_0
  - pygments=2.7.3=pyhd3eb1b0_0
  - pykerberos=1.2.1=py37h680d80a_2
  - pymongo=3.11.2=py37h2531618_0
  - pyopenssl=20.0.1=pyhd3eb1b0_1
  - pyparsing=2.4.7=py_0
  - pysocks=1.7.1=py37_1
  - pytest=6.2.1=py37h06a4308_0
  - pytest-forked=1.3.0=py_0
  - pytest-xdist=2.2.0=pyhd3eb1b0_0
  - python=3.7.9=h7579374_0
  - python-avro=1.9.2.1=py37hc8dfbb8_1
  - python-dateutil=2.8.1=py_0
  - python-hdfs=2.5.8=py_0
  - python_abi=3.7=1_cp37m
  - pytz=2020.5=pyhd3eb1b0_0
  - pyyaml=5.3.1=py37h7b6447c_1
  - re2=2020.11.01=h2531618_1
  - readline=8.0=h7b6447c_0
  - requests=2.25.1=pyhd3eb1b0_0
  - requests-kerberos=0.12.0=py37_0
  - rsa=4.7=pyhd3eb1b0_0
  - setuptools=51.1.2=py37h06a4308_3
  - sh=1.14.1=py37h06a4308_0
  - six=1.11.0=py37_1
  - snappy=1.1.8=he6710b0_0
  - sqlite=3.33.0=h62c20be_0
  - thrift-cpp=0.11.0=h02b749d_3
  - tk=8.6.10=hbc83047_0
  - toml=0.10.1=py_0
  - traitlets=5.0.5=py_0
  - typing-extensions=3.7.4.3=0
  - typing_extensions=3.7.4.3=py_0
  - uriparser=0.9.3=he6710b0_1
  - urllib3=1.26.2=pyhd3eb1b0_0
  - wcwidth=0.2.5=py_0
  - wheel=0.36.2=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - yaml=0.2.5=h7b6447c_0
  - zipp=3.4.0=pyhd3eb1b0_0
  - zlib=1.2.11=h7b6447c_3
  - zstd=1.3.7=h0b5b093_0
prefix: /opt/conda/envs/py3