Unable to import lxml etree on aws lambda

Question

{
  "errorMessage": "Unable to import module 'lambda_function': 
        cannot import name   'etree' from 'lxml' (/var/task/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

Also tried https://gist.github.com/allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639:

{
  "errorMessage": "Unable to import module 'lambda_function': 
     cannot import name 'etree' from 'lxml' 
     (/var/task/lxml-4.3.4-py3.6-linux-x86_64.egg/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

I am running on Ubuntu 18.04 on my local machine, and have also tried using the "Amazon Linux" image on an ec2 instance to build the bundle.

I have also tried, while within the activated venv:

STATIC_DEPS=true pip3 install lxml --target ./package --upgrade --no-cache-dir

I have also tried copying the shared object files based on pulling which files were opened when running the script with strace:

#! /bin/bash

export Z=$(pwd)/ok-daily-lambda.zip
rm $Z
zip $Z lambda_function.py
zip $Z __init__.py

for dir in $(find venv_here/lib/python3.6/site-packages)
do
    if [ -d $dir ] ; then
        pushd $dir
        echo zip -r9 $Z $(pwd)
        zip -r9 $Z $(pwd)
        popd
    fi
done

export LIBD=$(pwd)/lib
mkdir -p $LIBD

cp "/home/jmunsch/.local/lib/python3.6/site-packages/.libs_cffi_backend/libffi-d78936b1.so.6.0.4" $LIBD
cp "/lib/x86_64-linux-gnu/libbz2.so.1.0" $LIBD
cp "/lib/x86_64-linux-gnu/libc.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libdl.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libexpat.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libgcc_s.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/liblzma.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libm.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_dns.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_files.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libpthread.so.0" $LIBD
cp "/lib/x86_64-linux-gnu/libresolv.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/librt.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libtinfo.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libudev.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libutil.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libuuid.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libz.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libffi.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/liblz4.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libmpdec.so.2" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libssl.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libstdc++.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libzstd.so.1" $LIBD

zip -r $Z $LIBD

AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="xxx" AWS_DEFAULT_REGION="us-east-1" aws lambda update-function-code --function-name ok-today --zip-file fileb://ok-daily-lambda.zip

Here is the directory structure of the most recent zip file:

• https://gist.github.com/allen-munsch/5ddc27873e64db6ff106ab828febf434

Related:

• AWS Lambda not importing LXML
• LXML: Cannot import etree
• https://lxml.de/installation.html
• https://lxml.de/build.html
• lxml cannot import element etree
• Shared Libraries in Java AWS Lambda Project
• https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Answer 1

There are modules that cannot be added directly into the site-packages directory to be recognised inside an AWS Lambda environment. When that happens, you have to get an Amazon Linux image from Docker repositories and make your own compiled environment in a container version that will run on AWS Lambda

For example, if you want to use Python 3.6 a good choice will be amazonlinux:2018.03 in case you want to install more packages e.g. pandas, numpy, scipy

    docker run -v $(pwd):/outputs -it amazonlinux:2018.03

Since Amazon Linux is based on Red Hat, you have to install via yum all dependencies when running docker and having already created your virtual environment

    yum update -y
    yum install -y \
      python36 \
      python36-devel \
      python36-virtualenv \
      python34-setuptools \
      gcc \
      gcc-c++ \
      findutils \
      rsync \
      Cython \
      findutils \
      which \
      gzip \
      tar \
      man-pages \
      man \
      wget \
      make \
      zip

For lxml you will also need

    (lambda_docker) bash-4.2# yum install libxml2
    ...
    (lambda_docker) bash-4.2# yum install libxslt
    ...

You install the module as usual

    pip3.6 install lxml

You should see something like

    (lambda_docker) bash-4.2# pip3.6 install lxml
    Collecting lxml
      Downloading https://files.pythonhosted.org/packages/2d/53/34a9f0c79c548e430148837892b6ae91adee571a0e8b6c17bd7ff9c2d12e/lxml-4.3.4-cp36-cp36m-manylinux1_x86_64.whl (5.7MB)
         |################################| 5.7MB 2.0MB/s 
    Installing collected packages: lxml

Then, create your function lambda_function.py and add it in the zipped package pushd-ed and popd-ed from you docker session

    from lxml import etree

    def lambda_handler(event, context):
        print(__name__)
        print(etree.LXML_VERSION)

After created

    zip -g site-packages.zip lambda_function.py

Before uploading, you can check that your zip file contains the lxml directories

    [jonathan@docker lambda_docker]$ unzip -l site-packages.zip 
    Archive:  site-packages.zip
      Length      Date    Time    Name
    ---------  ---------- -----   ----
            0  06-29-2019 23:09   __pycache__/
          277  06-29-2019 23:09   __pycache__/easy_install.cpython-36.pyc
          126  06-29-2019 23:09   easy_install.py
          119  06-29-2019 23:29   lambda_function.py
            0  06-29-2019 23:21   lib/
            0  06-29-2019 23:39   lxml/
            0  06-29-2019 23:37   lxml-4.3.4.dist-info/
            4  06-29-2019 23:37   lxml-4.3.4.dist-info/INSTALLER
         2954  06-29-2019 23:37   lxml-4.3.4.dist-info/METADATA
        13384  06-29-2019 23:37   lxml-4.3.4.dist-info/RECORD
          109  06-29-2019 23:37   lxml-4.3.4.dist-info/WHEEL
            5  06-29-2019 23:37   lxml-4.3.4.dist-info/top_level.txt
         7668  06-29-2019 23:37   lxml/ElementInclude.py
          551  06-29-2019 23:37   lxml/__init__.py
            0  06-29-2019 23:37   lxml/__pycache__/
         3331  06-29-2019 23:37   lxml/__pycache__/ElementInclude.cpython-36.pyc
    ...

Now upload the zip and create an empty test {} in your lambda function

Result

    START RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64 Version: $LATEST
    lambda_function
    (4, 3, 4, 0)
    END RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64
    REPORT RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64  Duration: 0.30 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 50 MB  

If you prefer an image

Perfectly ready for AWS Lambda

Hope it helps (: