Search code examples
pythonaws-lambdaserverless-framework

Serverless python requirements slim:true does nothing for dependency size

Brief

when using the following settings in serverless.yml

custom:
  pythonRequirements:
    dockerizePip: true
    slim: true
    zip: true

slim:true and slim:false result in the same file size. What can I do to decrease the file size of the zip file?

Full

I'm trying to get python code and it's dependencies onto an aws lambda function using the serverless framework. following the recommendations provided here to reduce the dependency size to comply with the 250MB lambda limit. in my serverless.yml file I have the following:

provider:
  name: aws
  runtime: python3.6

plugins:
  - serverless-python-requierments

custom:
  pythonRequierments:
    dockerizePip: true
    slim: true
    zip: true

functions:
...

my requierments.txt is the following:

xgboost==1.3.3
pandas==1.0.1
numpy == 1.18.5
...

I keep getting the following error:

Unzipped size must be smaller than 262144000 bytes..

which leads me to believe that my requierments.zip file is too large; the verbose output stated uploading service test.zip file to s3 271.17 MB.

I was playing around with making the file smaller, and I ran sls deploy with dockerizePip, and slim set to false for testing purpose, and I got the same result; a test.zip file of 271.17 Mb. This seemed perplexing, as I understood it slim: true would reduce the file size prior to zipping the file, but running with slim:true and slim:false resulted in the same file size. How can I make slim work?

Update1:

I tried adding a slimPattern as discussed here, within pythonRequierments

slimPatterns:
  - '**/*.dist-info/**'

but all dist info directories still exist in the zip file, and the output is the same size (271.17MB)

Update 2:

I kept everything the same from update 1, but removed the .serverless directory and requierments.zip. All the .dist-info directories remain, even though I excluded them in slimPatterns (or so I thought).

Update 3:

I tried using package exclude, doing the following:

package:
  exclude:
    -node_modules/**
    -'**/*.dist-info/**'

this did succeed in getting node_modules removed (which I should have done before), but did not succeed in changing .requierments.zip in any way. all .dist-info directories remain.

Update 4:

I followed this, and added a noDeploy field in pythonRequirements, which included setuptools and six. despite this, both ended up in my .requirements.zip, and my file size hasn't changed (269.9 Mb, the same as update 3, which was slightly smaller due to the removal of node_modules)

Solution

  • It didn't work for me either. I ended up using Layers using the following option -

    custom:
  pythonRequirements:
    slim: true
    layer: true

    This reduced my deployment package size to less than 10mb. All the dependencies then go into a separate zip (~100mb) which creates a Lambda Layer. This helps reduce the Lambda cold-start problem as well.