I have a lambda function runs on Python 3.7 with architecture x86_64 before. Now I would like to migrate it to arm64 to use the Graviton processor and upgrade to Python 3.9 as well.
While I success to create the Python 3.9 virtual environment layer with the dependencies that I need, which is aws-encryption-sdk, when I change the architecture of my lambda function to arm64 and runtime to Python 3.9, below error shows after I test my code:
Unable to import module 'encryptor-lambda': /opt/python/cryptography/hazmat/bindings/_rust.abi3.so: cannot open shared object file: No such file or directory",
I went and check my virtual env layer and pretty sure the file /opt/python/cryptography/hazmat/bindings/_rust.abi3.so is existed there.
Then I tried to keep my runtime at Python 3.9 and switched back the architecture to x86, it works! Only if I try to change it to arm64, it has that error above.
I look up online and can't seems to have a solution or as of why is that. Is it not possible to migrate for the lambda functions that requires dependencies? Or am I missing anything?
Libraries like aws-encryption-sdk-python sometimes contain code/dependencies that are not pure Python and need to be compiled. When code needs to be "compiled" it is usually compiled for a target architecture (like ARM or x86) to run properly.
You can not run code compiled for one architecture on different architecture. So I suspect that is the reason for your error.
Looking at the error message I suspect it is the cryptography library causing this issue.
The library uses Rust. If you check your error, you will see that the shared library for the Rust binding is the causing your error (_rust.abi3.so). According to the documentation of the library, the ARM architecture is supported.
Therefore, I suspect that the way you are packaging your Lambda deployment package and it's dependencies is the issue. You are probably doing that on a computer with x86 architecture. Package manager like pip usually detect the OS and architecture they are run on and download dependencies for those OS's and architectures.
So I guess you have two options:
Both options are not really great.
Unfortunately, this is one of those areas where Python Lambdas can become very cumbersome to develop/deploy. Every time a depdency uses a non-Python extension (like a C extension), packaging/deployment becomes a problem.
Maybe someone else has a great tool to recommend.