Search code examples
pythonaws-lambdaparamikopysftp

Unable to import pysftp / paramiko through AWS lambda (facing different errors)

When a file is uploaded to s3 bucket, we need to process it in AWS lambda, gzip the processed file and send it some remote machine. We are able to process it and able to gzip the file. We thought of using pysftp or paramiko to send the processed zip file to remote machine. But we are facing some issues as below.

When we package pysftp / paramiko libraries along with our source code, deploy it on aws lambda, it is throwing error "Unable to import module 'lambda_function': No module named 'paramiko’”, / "Unable to import module 'lambda_function': No module named 'pysftp’”,

I am able to see the the libraries related to pysftp / paramiko libraries in my project path (in env/lib/python3.7/site-packages/) (I am using virtual env). The zip file I am uploading to lambda has the top directories as ‘env’ and my python source file. ‘env’ has the following structure (env -> lib -> python3.7 -> site-packages -> all python libraries)

enter image description here

 Error screenshot as below.

enter image description here

I saw in aws document that external libraries should be in python as top directory. So I created ‘python’ directory in my project and did ‘pip3 install pysftp -t python’. All the libraries related to pysftp are now in python directory. I created zip file (including env, python and my project source code) and zip file has the following structure

Zip -> env, python, lambda_function.py env -> lib -> python3.7 -> site-packages -> python distros, bot, bot core … python -> pysftp, paramkio, cryptography related …

Screenshot as below.

enter image description here

enter image description here

But I am getting same error "Unable to import module 'lambda_function': No module named 'paramiko’”, / "Unable to import module 'lambda_function': No module named 'pysftp’”.

There are some suggestions on internet to create aws layers, upload the zip file (pysftp or paramiko related) to layers and add the layer to aws lambda. Tried that. I created ‘python’ directory in my project and did ‘pip3 install pysftp -t python’. All the libraries related to pysftp are now in python directory. I created zip file of python (which includes pysftp, paramiko and cryptography related …) added that zip file to aws layer. Created one more zip file consists of env (lib -> python3.7 -> site-packages -> python distros, bot, bot core …) and source code. Uploaded this zip file to aws lambda. Added the layer to aws lambda. Now getting different error as below.

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /opt/python/cryptography/hazmat/bindings/_rust.abi3.so)

Error screenshot

enter image description here

On some of the posts, it was mentioned to uninstall cryptography and install cryptography 3.4.8.

But getting the same error.

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /opt/python/cryptography/hazmat/bindings/_rust.abi3.so)

I tried to do scp (instead of pysftp or paramiko) also to transfer the processed file to remote machine from aws lambda. But it failed with error ‘/bin/sh: scp: not found’

Can any one please let me know how to fix the issue (need to send the zip file to remote machine from lambda function)

Solution

  • I fixed the issue. Right now my zip file (to upload to the aws lambda) is looking below which is wrong.

    enter image description here

    zip -> env -> lib -> python3.7 -> site-packages -> python modules (this is current structure).

    Now I changed as below which is working.

    zip -> python modules + our source code.