Search code examples
pythonpython-2.7aws-lambdapymysql

import from a folder with dots in name - Python

I have a python package which looks like the following:

package/
├── __init__.py
├── PyMySQL-0.7.6-py2.7.egg
├── pymysql
├── PyMySQL-0.7.x.pth  
└── tests.py

The folder structure cannot be changed because it is from a third party library.

The contents of the .pth file are 

import sys; sys.__plen = len(sys.path)
./PyMySQL-0.7.6-py2.7.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Whats the best way to include pymysql in tests.py

I obviously can't use from PyMySQL-0.7.6-py2.7.egg because the folder name contains dots.

P.S. absolute paths are not known because this code is supposed to be deployed to AWS lambda

Solution

  • The problem is that the module's .pth file is not being used. Amazon's build instructions for Lambda have you install your dependencies into the same root directory as your Lambda function file. This directory is not configured as a "site directory" (e.g. like lib/pythonX.Y/site-packages/ in a virtualenv, or for global installations, /usr/lib/pythonX.Y/site-packages (CentOS) or /usr/local/lib/pythonX.Y/dist-packages (Ubuntu)), and so .pth files are not evaluated.

    You can configure the current working directory as a site directory. The site module documentation describes creating a usercustomize.py as the convention for adding custom site directories, but Lambda presents another hiccup: to be evaluated automatically, the file is expected to be found in ~/.local/lib/pythonX.Y/site-packages. I created a usercustomize.py in the root of my Lambda .zip file and imported it first in my Lambda function handler file. I was then able to import modules that use .pth files to configure dot notation.

    usercustomize.py

    import site
import os

site.addsitedir(os.getcwd())

    lambda_handler.py

    import usercustomize
from foo.bar import baz

    Using this method, you could install the dependencies into a site-packages dir in the root of your distribution zip file (os.getcwd() becomes os.path.join(os.getcwd(), 'site-packages')), making the structure a little cleaner, if that matters to you.