Search code examples
aws-lambdaamazon-efs

Importing LightGBM into AWS Lambda

I currently have an AWS Lambda function that's connected to an EFS. The EFS is storing all of my dependencies, such as numpy, pandas, and lightgbm. I have no trouble importing any package except lightgbm. When I do, it raises the error:

"libgomp.so.1: cannot open shared object file: No such file or directory"

How do I fix it?

I've tried uploading LightGBM as a lambda layer, as well as packaging libgomp into the lambda layer, but neither have worked.

Solution

  • After much effort and numerous attempts over the course of two days, I was able to resolve the issue. It's worth noting that the versions of libgomp.so and glibc can vary depending on the Amazon Lambda runtime version. For more information, refer to https://repost.aws/questions/QUrXOioL46RcCnFGyELJWKLw/glibc-2-27-on-amazon-linux-2. enter image description here To address this, you can create a custom AWS Lambda layer by provisioning an Amazon Linux 2 instance on an EC2, downloading the required libgomp.so and glibc packages, and packaging them into a Lambda layer. Ensure that the AWS Lambda layer objects are located under the /lib folder, as specified in the documentation: https://docs.aws.amazon.com/lambda/latest/dg/packaging-layers.html.

    Once this layer is added, the Lightgbm library should be importable in your Lambda environment.

    here my layer for python 3.8! AWS-Lambda-Layers-v0.zip