Search code examples
pythoncloud-foundryreticulate

Python with --enable-shared on cloudfoundry

I would like to deploy an application on a cloudfoundry instance, consisting of an Rmarkdown file which should contain both R and python chunks. I'd like to render the document on cloudfoundry, rather than doing so locally and just pushing up the HTML file.

For knitr to work with python, it uses the reticulate package which states in the docs:

Note that for reticulate to bind to a version of Python it must be compiled with shared library support (i.e. with the --enable-shared flag).

I tried to use the cloudfoundry python buildpack to provide python for reticulate, but got the error Python shared library not found, Python bindings not loaded, and reticulate::py_discover_config() on the cloudfoundry app gives:

> reticulate::py_discover_config()
python:         /home/vcap/deps/2/bin/python/  # or something very like this
libpython:      [NOT FOUND]

which I think suggests that python hasn't been compiled with --enable-shared.

Tantalisingly, the changelog has this line * Add --enable-shared to Python runtime options from 6 years ago, but it looks to be for a specific version that is no longer relevant.

Is there a way I can get a version of python on cloudfoundry which will work with reticulate? Ideally without having to build it from scratch myself...

Solution

  • I have found that I can install a version of python that reticulate is happy to work with, using reticulate itself. Ahead of rendering the Rmarkdown document, I can use:

    reticulate::install_miniconda()
reticulate::py_install(readLines("requirements.txt"))

    This creates a conda environment with all my requirements and allows my code to run.

    The disadvantage is that it shifts the python installation from the app staging (which has 15 minutes to complete by default on cloudfoundry), to the app start command (which has 60s by default). I had to increase the default timeout to 120s to give enough time to install the requirements and render the document.

    Following @DanielMikusa's comment, I have also raised a feature request on the python buildpack, to hopefully enable shared library support in future, but this works for now for me.