I am working on a project which uses the CircleCI continuous integration platform. I use Python as main language and Miniconda as platform. I want to test multiple Python and NumPy versions using Miniconda on CircleCI.
I tried to use different Python images but it uses only Python 3.7 as I install the latest Miniconda version. Could you tell me how to use multiple versions?
Below is config.yml
:
version: 2.0
workflows:
version: 2
test:
jobs:
- py3.6-np1.15
- py3.5-np1.15
- py3.6-np1.14
- py3.5-np1.14
- py3.7-np1.15
- py3.5-np1.16
- py3.6-np1.16
- py3.7-np1.16
jobs:
py3.6-np1.15: &test-template
docker:
- image: circleci/python:3.6.8
environment:
NUMPY_VERSION: 1.15.2
CYTHON_VERSION: 0.29.2
working_directory: ~/repo
steps:
- checkout
- run:
name: Install System Dependencies
command: sudo apt-get update && sudo apt-get install -y libmpich12 libmpich-dev build-essential
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum "setup.py" }}
- run:
name: install anaconda
command: |
wget https://repo.continuum.io/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh -O ~/miniconda.sh
chmod +x ~/miniconda.sh && ~/miniconda.sh -b
export PATH=$HOME/miniconda3/bin:$PATH
conda update --quiet --yes conda
- run:
name: Install numpy, cython, mdtraj
command: |
export PATH=$HOME/miniconda3/bin:$PATH
conda install --quiet --yes numpy==$NUMPY_VERSION cython==$CYTHON_VERSION
conda install --quiet --yes -c conda-forge mdtraj
# - run:
# name: Upgrade pip
# command: |
# python3 -m venv venv
# . venv/bin/activate
# pip install pip==18.0
# - run:
# name: Install numpy and cython
# command: |
# python3 -m venv venv
# . venv/bin/activate
# pip install --progress-bar off numpy==$NUMPY_VERSION cython==$CYTHON_VERSION
- run:
name: Install and build
command: |
export PATH=$HOME/miniconda3/bin:$PATH
pip install --progress-bar off .[dev]
python setup.py build_ext --inplace
python setup.py install
py3.5-np1.15:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.6-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.5-np1.14:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.29.2
py3.7-np1.15:
<<: *test-template
docker:
- image: circleci/python:3.7.3
py3.5-np1.16:
<<: *test-template
docker:
- image: circleci/python:3.5.7
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
py3.6-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
py3.7-np1.16:
<<: *test-template
docker:
- image: circleci/python:3.7.3
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
Here is a minimal example configuration on how to use CircleCI with Miniconda and specific Python and NumPy versions, starting with an empty ubuntu:bionic
image.
version: 2
jobs:
build:
docker:
- image: ubuntu:bionic
environment:
PYTHON_VERSION: 3.5.5
NUMPY_VERSION: 1.14.2
steps:
- checkout
- run:
name: Setup Miniconda
command: |
apt update
apt install -y wget
cd $HOME
wget "https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh" -O miniconda.sh
printf '%s' "8a324adcc9eaf1c09e22a992bb6234d91a94146840ee6b11c114ecadafc68121 miniconda.sh" | sha256sum -c
bash miniconda.sh -b -p $HOME/miniconda
- run:
name: Setup environment and run tests
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update -y conda
conda create -n myenv python=$PYTHON_VERSION -c conda-forge
source activate myenv
conda install -y numpy=$NUMPY_VERSION
python --version
python -c "import numpy; print(numpy.__version__)"
I think it's good practice to verify the checksum after downloading the Miniconda install script Miniconda3-4.7.10-Linux-x86_64.sh
from the internet.
You can change environment variables PYTHON_VERSION
and NUMPY_VERSION
to get other versions.
Instead of "real" tests, currently we're just going to verify that our desired versions for Python and NumPy are being used with python --version
and python -c "import numpy; print(numpy.__version__)"
. For the example above, at the end of the log you should find:
Python 3.5.5
1.14.2
Depending on the versions you select, you might get an error:
PackagesNotFoundError
, you need to make sure that the selected channel has the package version you're looking for. (Like conda-forge
is selected in the example above.)UnsatisfiableError
, you have selected versions of the packages which aren't compatible.Here is an example configuration for multiple versions:
version: 2
workflows:
version: 2
test:
jobs:
- python_3.5
- python_3.6
- python_3.7
template: &template
docker:
- image: ubuntu:bionic
steps:
- checkout
- run:
name: Setup Miniconda
command: |
apt update
apt install -y wget
cd $HOME
wget "https://repo.anaconda.com/miniconda/Miniconda3-4.7.10-Linux-x86_64.sh" -O miniconda.sh
printf '%s' "8a324adcc9eaf1c09e22a992bb6234d91a94146840ee6b11c114ecadafc68121 miniconda.sh" | sha256sum -c
bash miniconda.sh -b -p $HOME/miniconda
- run:
name: Setup environment and run tests
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update -y conda
conda create -n myenv python=$PYTHON_VERSION
source activate myenv
conda install -y pip numpy=$NUMPY_VERSION
python --version
pip --version
python -c "import numpy; print(numpy.__version__)"
jobs:
python_3.5:
<<: *template
environment:
PYTHON_VERSION: 3.5
NUMPY_VERSION: 1.14.2
python_3.6:
<<: *template
environment:
PYTHON_VERSION: 3.6
NUMPY_VERSION: 1.15.2
python_3.7:
<<: *template
environment:
PYTHON_VERSION: 3.7
NUMPY_VERSION: 1.16.5
If I apply this minimal example to your case, the configuration would look similar to this:
version: 2.0
workflows:
version: 2
test:
jobs:
- py3.6-np1.15
- py3.5-np1.15
- py3.6-np1.14
- py3.5-np1.14
- py3.7-np1.15
- py3.6-np1.16
- py3.7-np1.16
test-template: &test-template
docker:
- image: ubuntu:bionic
steps:
- checkout
- run:
name: Install System Dependencies
command: apt-get update && apt-get install -y libmpich12 libmpich-dev build-essential
# Download and cache dependencies
- restore_cache:
keys:
- v1-dependencies-{{ .Environment.CIRCLE_JOB }}-{{ checksum "setup.py" }}
- run:
name: install anaconda
command: |
apt update
apt install -y wget
cd $HOME
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
chmod +x ~/miniconda.sh && bash ~/miniconda.sh -b -p $HOME/miniconda
export PATH=$HOME/miniconda/bin:$PATH
- run:
name: Install numpy, cython, mdtraj
command: |
export PATH="$HOME/miniconda/bin:$PATH"
conda update --yes conda
echo $PYTHON_VERSION
conda create -n myenv python=$PYTHON_VERSION -c conda-forge
source activate myenv
conda install --yes pip
conda install --yes -c conda-forge numpy=$NUMPY_VERSION cython=$CYTHON_VERSION
conda install --yes -c conda-forge nose mdtraj
python --version
python -c "import numpy; print(numpy.__version__)"
- run:
name: Install and build package
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
pip install --progress-bar off .[dev]
python setup.py build_ext --inplace
python setup.py install
- save_cache:
paths:
- ~/miniconda
key: v1-dependencies-{{ checksum "setup.py" }}
- run:
name: Run non-MPI tests
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
nosetests -a '!mpi' package
- run:
name: Run MPI tests
command: |
export PATH=$HOME/miniconda/bin:$PATH
source activate myenv
OMP_NUM_THREADS=1 mpiexec -n 2 nosetests -a mpi package
- store_artifacts:
path: test-reports
destination: test-reports
jobs:
py3.6-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.5-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.5
py3.6-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.5-np1.14:
<<: *test-template
environment:
NUMPY_VERSION: 1.14.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.5
py3.7-np1.15:
<<: *test-template
environment:
NUMPY_VERSION: 1.15.2
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.7.1
py3.6-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.26.1
PYTHON_VERSION: 3.6
py3.7-np1.16:
<<: *test-template
environment:
NUMPY_VERSION: 1.16.5
CYTHON_VERSION: 0.29.2
PYTHON_VERSION: 3.7.1