Serverless python packaging numpy dependency error

Question

I have been running into issues when making function calls from my deployed Python3.7 Lambda function that, from the error message, are related to numpy. The issue states that there is an inability to import the package and despite trying many of the solutions I have read about, I haven't had any success and I am wondering what to test out next or how to debug further.

I have tried the following:

1. Install Docker, add the plugin serverless-python-requirements, configure in yml
2. Install packages in app directory to be bundled and deployed, pip install -t src/vendor -r requirements.txt --no-cache-dir
3. Uninstalled setuptools and numpy and reinstalled in that order

Error Message (Displayed after running sls invoke -f auth):

{
    "errorMessage": "Unable to import module 'data': Unable to import required dependencies:\nnumpy: \n\nIMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!\n\nImporting the numpy c-extensions failed.\n- Try uninstalling and reinstalling numpy.\n- If you have already done that, then:\n  1. Check that you expected to use Python3.7 from \"/var/lang/bin/python3.7\",\n     and that you have no directories in your PATH or PYTHONPATH that can\n     interfere with the Python and numpy version \"1.18.1\" you're trying to use.\n  2. If (1) looks fine, you can open a new issue at\n     https://github.com/numpy/numpy/issues.  Please include details on:\n     - how you installed Python\n     - how you installed numpy\n     - your operating system\n     - whether or not you have multiple versions of Python installed\n     - if you built from source, your compiler versions and ideally a build log\n\n- If you're working with a numpy git repository, try `git clean -xdf`\n  (removes all files not under version control) and rebuild numpy.\n\nNote: this error has many possible causes, so please don't comment on\nan existing issue about this - open a new one instead.\n\nOriginal error was: No module named 'numpy.core._multiarray_umath'\n",
    "errorType": "Runtime.ImportModuleError"
}

Provided is my setup:

OS: Mac OS X
Local Python: /Users/me/miniconda3/bin/python
Local Python version: Python 3.7.4

Serverless Environment Information (Runtime = Python3.7):

 Operating System:          darwin
 Node Version:              12.14.0
 Framework Version:         1.67.3
 Plugin Version:            3.6.6
 SDK Version:               2.3.0
 Components Version:        2.29.1

Docker:

Docker version 19.03.13, build 4484c46d9d

serverless.yml:

service: understand-your-sleep-api

plugins:
  - serverless-python-requirements
  - serverless-offline-python

custom:
  pythonRequirements:
    dockerizePip: true # non-linux
    slim: true
    useStaticCache: false
    useDownloadCache: false
    invalidateCaches: true

provider:
  name: aws
  runtime: python3.7
  stage: ${opt:stage, 'dev'}
  region: us-east-1
  iamRoleStatements:
    - Effect: Allow
      Action:
        - ssm:GetParameter
      Resource: "arn:aws:ssm:us-east-1:*id*:parameter/*"
  environment:
    STAGE: ${self:provider.stage}

functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true

package:
  exclude:
    - env.yml
    - node_modules/**

requirements.txt:

pandas==1.0.0
fitbit==0.3.1
oauthlib==3.1.0
requests==2.22.0
requests-oauthlib==1.3.0

data.py:

import sys
sys.path.insert(0, 'src/vendor') # Location of packages that follow
import json
from datetime import timedelta, datetime, date
import math
import pandas as pd
from requests_oauthlib import OAuth2Session
from urllib.parse import urlparse, parse_qs
import fitbit
import requests
import webbrowser
import base64
import os 
import logging

def auth(event, context):
    ...

Answer 1

• Use lambda layer to pack all your requirements, Make sure you have numpy in requirements.txt file. Try it once.
• This works only when serverless-python-requirements plugin is listed in plugins section.
• Replace your custom key with this and give the functions a reference to use that layer

custom:
  pythonRequirements:
    layer: true
functions:
  auth:
    handler: data.auth
    events:
      - http:
          path: /auth
          method: get
          cors: true
    layers:
      - { Ref: PythonRequirementsLambdaLayer}