Search code examples
pythonaws-lambdadependenciespypi

How do I download WHL files more efficiently?

I'm trying to use the pandasai library on AWS Lambda via Lambda layers, but since my local machine is using Windows, I'm downloading the Linux WHL file of the library from PyPI to put as a Lambda layer. However, since pandasai requires me to install a lot of co-dependencies in the same way (matplotlib, pandas, etc.), it will take a really long time.

Is there a quicker/better way to do this? Are there more efficient ways to install Linux architecture python libraries? (Like using Docker images/VMs)

Thanks.

Solution

  • You can use the below command in a writable directory as quicker/better way:

    mkdir python
cd python
pip install --platform manylinux2014_x86_64 --target=.\ --implementation cp --python-version 3.8 --only-binary=:all: --upgrade pandasai
    1. Creates a folder
    2. Move to the created folder
    3. Install the package with pip specifying the below attributes
    4. Finally zip the python folder and proceed to create the lambda layer.
    • —-platform: Specifies the platform on which the package should be installed.
    • —-target: Sets the installation target directory for the package.
    • —-implementation: Specifies the Python implementation to use during installation
    • —-python-version: Specifies the Python version for which the package should be installed.
    • —-only-binary: Instructs pip to only consider binary packages during installation.
    • —-upgrade: Tells pip to upgrade the package if it's already installed. If a previous version of the package exists, it will be updated to the latest version available.