Search code examples
rcondacranconda-build

conda build fails for CRAN R packages

I am trying to build a R conda package that doesn't exist on conda-forge from a CRAN package using the appraoch shown here: https://www.anaconda.com/blog/developer-blog/conda-data-science/

I used the R package rann as an example - this package already exists as r-rann on conda-forge, but I used it to test if the build process works in general.

During the build process the script "conda_build.sh" is executed that contains the line $R CMD INSTALL --build .. My problem is that the build process fails with error Fatal error: cannot open file 'CMD': No such file or directory independently of which package I try to build. I have no idea where the environment variable(?) "CMD" should come from.

I took the following steps:

  1. conda skeleton cran rann
  2. conda build r-rann

I also tested to build Python packages from pip and these work without any problem, so it doesn't appear that this is a general problem with my Miniconda installation.

My system:

  • Ubuntu 17.10
  • Miniconda 3, version 4.4.10

Here is the full log of the build process:

eega@eega-nb:~/workspace$ conda build r-rann/
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for r-rann
INFO:conda_build.metadata:Attempting to finalize metadata for r-rann
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


BUILD START: ['r-rann-2.5.1-r341h599a50d_0.tar.bz2']
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold


The following NEW packages will be INSTALLED:

    bzip2:           1.0.6-1          conda-forge
    ca-certificates: 2018.1.18-0      conda-forge
    cairo:           1.14.10-0        conda-forge
    curl:            7.59.0-0         conda-forge
    fontconfig:      2.12.6-0         conda-forge
    freetype:        2.8.1-0          conda-forge
    gettext:         0.19.8.1-0       conda-forge
    glib:            2.55.0-0         conda-forge
    graphite2:       1.3.11-0         conda-forge
    gsl:             2.1-2            conda-forge
    harfbuzz:        1.7.6-0          conda-forge
    icu:             58.2-0           conda-forge
    jpeg:            9b-2             conda-forge
    krb5:            1.14.2-0         conda-forge
    libffi:          3.2.1-3          conda-forge
    libgcc:          7.2.0-h69d50b8_2            
    libgcc-ng:       7.2.0-hdf63c60_3            
    libiconv:        1.15-0           conda-forge
    libpng:          1.6.34-0         conda-forge
    libssh2:         1.8.0-2          conda-forge
    libstdcxx-ng:    7.2.0-hdf63c60_3            
    libtiff:         4.0.9-0          conda-forge
    libxml2:         2.9.8-0          conda-forge
    ncurses:         5.9-10           conda-forge
    openssl:         1.0.2n-0         conda-forge
    pango:           1.40.14-0        conda-forge
    pcre:            8.39-0           conda-forge
    pixman:          0.34.0-1         conda-forge
    r-base:          3.4.1-3          conda-forge
    readline:        7.0-0            conda-forge
    tk:              8.6.7-0          conda-forge
    xz:              5.2.3-0          conda-forge
    zlib:            1.2.11-0         conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda


Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.4.10
  latest version: 4.5.0

Please update conda by running

    $ conda update -n base conda



## Package Plan ##

  environment location: /tmp/conda-builds/r-rann_1522400915206/_build_env


The following NEW packages will be INSTALLED:

    binutils_impl_linux-64: 2.28.1-had2808c_3            
    binutils_linux-64:      7.2.0-26                     
    gcc_impl_linux-64:      7.2.0-habb00fd_3             
    gcc_linux-64:           7.2.0-26                     
    gxx_impl_linux-64:      7.2.0-hdf63c60_3             
    gxx_linux-64:           7.2.0-26                     
    libstdcxx-ng:           7.2.0-hdf63c60_3             
    make:                   4.2.1-0           conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Source cache directory is: /tmp/conda-builds/src_cache
Found source in cache: RANN_2.5.1_75277e5d8a.tar.gz
Extracting download
source tree in: /tmp/conda-builds/r-rann_1522400915206/work
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-addr2line
+AR=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ar
+AS=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-as
+CXXFILT=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++filt
+ELFEDIT=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-elfedit
+GPROF=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gprof
+HOST=x86_64-conda_cos6-linux-gnu
+LD_GOLD=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ld.gold
+LD=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ld
+NM=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-nm
+OBJCOPY=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-objcopy
+OBJDUMP=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-objdump
+RANLIB=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-ranlib
+READELF=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-readelf
+SIZE=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-size
+STRINGS=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-strings
+STRIP=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+CC=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2
+CPP=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-cpp
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og
+GCC_AR=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc-ar
+GCC_NM=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc-nm
+GCC_RANLIB=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc-ranlib
+GCC=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-gcc
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib -L/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+CXX=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-c++
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -pipe -I/tmp/conda-builds/r-rann_1522400915206/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+GXX=/tmp/conda-builds/r-rann_1522400915206/_build_env/bin/x86_64-conda_cos6-linux-gnu-g++
Fatal error: cannot open file 'CMD': No such file or directory
Traceback (most recent call last):
  File "/home/eega/miniconda3/bin/conda-build", line 6, in <module>
    sys.exit(conda_build.cli.main_build.main())
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main
    execute(sys.argv[1:])
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute
    verify=args.verify)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/api.py", line 199, in build
    notest=notest, need_source_download=need_source_download, variants=variants)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 2097, in build_tree
    notest=notest,
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 1370, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir, stats=build_stats)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/utils.py", line 301, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/home/eega/miniconda3/lib/python3.6/site-packages/conda_build/utils.py", line 280, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/tmp/conda-builds/r-rann_1522400915206/work/conda_build.sh']' returned non-zero exit status 2.

The content of the build.sh conda created:

#!/bin/bash

if [[ $target_platform =~ linux.* ]] || [[ $target_platform == win-32 ]] || [[ $target_platform == win-64 ]] || [[ $target_platform == osx-64 ]]; then
  export DISABLE_AUTOBREW=1
  mv DESCRIPTION DESCRIPTION.old
  grep -v '^Priority: ' DESCRIPTION.old > DESCRIPTION
  $R CMD INSTALL --build .
else
  mkdir -p $PREFIX/lib/R/library/RANN
  mv * $PREFIX/lib/R/library/RANN
fi

The content of the meta.yaml conda created:

{% set version = '2.5.1' %}

{% set posix = 'm2-' if win else '' %}
{% set native = 'm2w64-' if win else '' %}

package:
  name: r-rann
  version: {{ version|replace("-", "_") }}

source:
  fn: RANN_2.5.1.tar.gz
  url:
    - {{ cran_mirror }}/src/contrib/RANN_2.5.1.tar.gz
    - {{ cran_mirror }}/src/contrib/Archive/RANN/RANN_2.5.1.tar.gz
  sha256: 75277e5d8a13ca01ff387f99d403268a8077862d4e95b076b74fb1b5538a8546

build:
  merge_build_host: True  # [win]
  # If this is a new build for the same version, increment the build number.
  number: 0

  # This is required to make R link correctly on Linux.
  rpaths:
    - lib/R/lib/
    - lib/

# Suggests: testthat
requirements:
  build:
    - {{ compiler('c') }}          # [not win]
    - {{ compiler('cxx') }}        # [not win]
    - {{native}}toolchain          # [win]
    - {{posix}}filesystem          # [win]
    - {{posix}}make

  host:
    - r-base

  run:
    - r-base
    - {{native}}gcc-libs           # [win]

test:
  commands:
    # You can put additional test commands to be run here.
    - $R -e "library('RANN')"           # [not win]
    - "\"%R%\" -e \"library('RANN')\""  # [win]

  # You can also put a file called run_test.py, run_test.sh, or run_test.bat
  # in the recipe that will be run at test time.

  # requires:
    # Put any additional test requirements here.

about:
  home: https://github.com/jefferis/RANN
  license: GPL (>= 3)
  summary: Finds the k nearest neighbours for every point in a given dataset in O(N log N) time
    using Arya and Mount's ANN library (v1.1.3). There is support for approximate as
    well as exact searches, fixed radius searches and 'bd' as well as 'kd' trees. The
    distance is computed using the L2 (Euclidean) metric. Please see package 'RANN.L1'
    for the same functionality using the L1 (Manhattan, taxicab) metric.
  license_family: GPL3

# The original CRAN metadata for this package was:

# Package: RANN
# Title: Fast Nearest Neighbour Search (Wraps ANN Library) Using L2 Metric
# Author: Sunil Arya and David Mount (for ANN), Samuel E. Kemp, Gregory Jefferis
# Maintainer: Gregory Jefferis <jefferis@gmail.com>
# Copyright: ANN library is copyright University of Maryland and Sunil Arya and David Mount. See file COPYRIGHT for details.
# Description: Finds the k nearest neighbours for every point in a given dataset in O(N log N) time using Arya and Mount's ANN library (v1.1.3). There is support for approximate as well as exact searches, fixed radius searches and 'bd' as well as 'kd' trees. The distance is computed using the L2 (Euclidean) metric. Please see package 'RANN.L1' for the same functionality using the L1 (Manhattan, taxicab) metric.
# URL: https://github.com/jefferis/RANN
# BugReports: https://github.com/jefferis/RANN/issues
# Encoding: UTF-8
# License: GPL (>= 3)
# Suggests: testthat
# Version: 2.5.1
# RoxygenNote: 6.0.1
# NeedsCompilation: yes
# Packaged: 2017-05-19 13:52:30 UTC; jefferis
# Repository: CRAN
# Date/Publication: 2017-05-21 07:56:50 UTC

# See
# http://docs.continuum.io/conda/build.html for
# more information about meta.yaml

Any help would be appreciated!

Solution

  • "CMD" is an R command-line instruction. During the conda-build you download the source code for RANN, then build a package from it using R in a temp environment, then install it (temporarily) to check it works.

    That line is just telling R to build the package rann. When you're developing an R package you use R CMD build <my_package> or R CMD INSTALL --build <my_package> to make an installable archive of that package.

    It looks like your conda-build can't find an R to build the package, although that is typically done in a temporary environment that is set up while running conda-build; and the build process will almost certainly install r-base so there's no real reason for it not to find R.

    I've ran the code under windows-subsystem for linux (effectively Ubuntu 16.04) with conda-build 3.7.2 and conda 4.3.34, and with R v3.4.1 (with verbose output)

    conda skeleton cran rann
conda build r-rann --R=3.4.1 --debug

    It tries to run Rscript CMD INSTALL --build on my system, rather than R CMD build. The former isn't how packages are typically built under R, and it was at this step that the "Cannot find file CMD" error was thrown, so I had a look into it in a bit more detail.

    First, I compared the meta.yaml with that on conda-forge. libgcc was absent from the run-requirements and r-base was absent from the build-requirements in the yaml set up by conda skeleton - I added these and rebuilt (though I doubt these cause the problem). This didn't fix anything, the "Could not find CMD" error still threw.

    I checked the changelog at conda-build. Rscript is used to run tests rather than R from version 3.7.2 onwards. I reverted to conda-build=3.7.1:

    conda install conda-build=3.7.1 -vv
conda build r-rann --R=3.4.1 --debug

    Now, I get past the ccall to R CMD INSTALL --build ..., but an error still occurs during compilation; so I haven't solved the problem yet.

    Nonetheless, it looks like the error you originally posted about has resulted from a change to conda-build (to use Rscript rather than R when testing packages) that has led to problems in building R packages. Could you post an issue cross-referencing this question to the conda-build github, please.

    <-- EDIT -->

    After some searching, I got this to work by a) installing gfortran and b) using conda-build 3.6.0 to both create the skeleton and build the package. 3.7.0/1 of conda-build threw a bug related to version numbers (without the --version XXX argument, conda-skeleton threw an error re Namespace not having a version; with the --version XXX argument, conda-skeleton threw an error re version numbers not being implemented for R packages).

    The above errors have been fixed in conda-build 3.8.0, but I haven't used that yet