Error importing a cffi python module from AWS Lambda Layer

Question

I am having a problem importing the argon2-cffi python module (version 20.1.0) from an AWS Lambda Layer into an AWS Lambda function.

This seems to apply to all packages that have C as a base, because I tested this with pandas and got the same results.

My cloudformation config for the Layer is the following:

  MyLib:
    Type: AWS::Serverless::LayerVersion
    Properties:
      LayerName: my-lib
      Description: In-house and 3rd party dependencies for my app.
      ContentUri: lambdas/lib/.
      CompatibleRuntimes:
        - python3.8

and the folder structure inside /lib is this:

lib
├── python
│   ├── app
│   │   └── ...
│   └── lib/python3.8/site-packages
│       ├── argon2
│       │   └── ...
│       ├── argon2_cffi-20.1.0.dist-info
│       │   └── ...
│       ├── ...
│       └── stdnum
│           └── ...
└── requirements.txt

as per recommendations from AWS, where the folder python/app contains my custom library and the folder python/lib/python3.8/site-packages contains 3rd party packages.

I know that since argon2 depends on C-code, it must be installed on the runtime environment that the Lambda function using it will be hosted in, so I use a docker image provided by AWS to install the packages into lib/python3.8/site-packages using the command

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

The lambda function I'm trying to run is configured in the cloudformation template as

  AuthorizerFunction:
    Type: 'AWS::Serverless::Function'
    Properties:
      CodeUri: lambdas/handlers
      Handler: authorizer.handler
      Runtime: python3.8
      ...
      Layers:
        - !Ref MyLib

and starts with these imports:

# authorizer.py

import app.utils as utils
import stdnum
from argon2 import PasswordHasher

But when I try to run it, I get the following error

[ERROR] Runtime.ImportModuleError: Unable to import module 'authorizer': No module named 'argon2._ffi'

meaning that app and stdnum both get imported successfully, but not argon2.

Any ideas what might be wrong?

I would also welcome any suggestions on another hashing library with a tried and tested hashing algorithm.

UPDATE: Updating my cffi, pip and setuptools as is recommended here did not do the trick.

UPDATE 2: I can see that the argon being installed using the docker command is the linux compatible one, since the output includes these lines:

Collecting argon2-cffi==20.1.0
  Downloading argon2_cffi-20.1.0-cp35-abi3-manylinux1_x86_64.whl (97 kB)

Answer 1

Alright, so the answer to my problem is a bit of an awkward one. I'll post it here anyhow in case it might depthen an understanding for someone.

I am using git for source control and used a template from VS Code for the .gitignore file. Furthermore, I have set up an automatic build and deployment process, which builds the Lambda application based on the content in the git repo.

After days of trying to debug this problem I found out that .so files were missing from the argon2_cffi folder in the Lambda layer. The culprit was the aforementioned .gitignore file, which contained these lines:

# C extensions
*.so

No wonder none of the C-based python packages were working!

I removed said lines from .gitignore and committed all the .so files to the repo that had never made it to the Lambda Layer before, and now everything is working fine.