Using custom dependencies in Google Cloud ML Engine

Question

I am trying to use a python package that is not listed in PyPI with the Google Cloud ML Engine. This package has itself dependencies that even though are listed in PyPI are not installed by default in the ML engine environment, namely the Cython package.

Looking at the documentation it is not really clear how to proceed in this case, I have tried packaging this package in a .tar.gz file and passing it under the --packages argument, but I got the following error:

File "<string>", line 1, in <module> IOError: [Errno 2] No such file or directory: '/tmp/pip-jnm3Ml-build/setup.py'

After I tried using a setup.py file and packaging my code but google cloud ml engine is not able to find the package in dependency_links

Here is my current setup.py:

from setuptools import find_packages, setup

required_packages = ['cython', 'numpy', 'tensorflow', 'scipy', 'cython']
dependency_links = ['git+https://github.com/lucasb-eyer/pydensecrf.git']

setup(name='trainer',
      version='0.1',
      packages=['trainer'],
      install_requires=required_packages,
      dependency_links=dependency_links,
      include_package_data=True,
      description='description')

I would like to avoid doing this by trial and error since sending jobs to the cloud costs money even if they fail immediately.

Thanks in advance.

Answer 1

To do this, you will need to add the Cython to the list of required packages in your setup.py. Instructions can be found here.

Here is a sample setup.py, that would reside in the parent directory of the directory you pass as --package-path to gcloud.

from setuptools import find_packages
from setuptools import setup

REQUIRED_PACKAGES = ['Cython>=0.26']

setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My trainer application package.'
)