How do I get relative package.json dependencies to work with AWS's sam build command on Windows?

Question

My goal is to share library code among several of our lambda functions using layers and be able to debug locally and run tests.

npm is able to install dependencies from the local file system. When we change our library code, we want all users of that library to get the updated code without having to set up a dedicated npm server. I'm able to debug locally just fine using the relative paths, but that's before I involve sam build. sam build creates a hidden folder at the root of the repository and builds the folder out and eventually runs npm install, however this time the folder structure is different. The relative paths used in the package.json file are now broken. We can't use explicit paths because our repositories reside under our user folders, which are of course different from one developer to another.

Here's what I did:

I created a project using sam init and took the defaults (except the name of sam-app-2) for a nodejs 12.x project (options 1 and 1).

That command created a folder called sam-app-2 which is the reference for all of the following file names.

I created a dependencies/nodejs folder.

I added dep.js to that folder:

exports.x = 'It works!!';

I also added package.json to the same folder:

{
  "name": "dep",
  "version": "1.0.0",
  "description": "",
  "main": "dep.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Under hello-world (the folder housing the lambda function), I added the following to the dependencies in package.json:

"dep": "file:../dependencies/nodejs"

I ran npm install under hello-world and it copied the dependencies under node_modules/dep. Normally, you would't do that here. This is purely to allow me to run locally without involving the sam CLI. It's just pure nodejs code. I can run tests, I can debug and not have to wait twenty seconds or more while sam packages up everything and invokes my function. Developing in this state is awesome because it's very fast. However, it'll eventually need to run correctly in the wild.

I edited ./hello-world/app.js:

const dep = require('dep');
let response;

exports.lambdaHandler = async (event, context) => {
    try {
        // const ret = await axios(url);
        response = {
            'statusCode': 200,
            'dep.x': dep.x,
            'body': JSON.stringify({
                message: 'Hello, World!!',
                // location: ret.data.trim()
            })
        }
    } catch (err) {
        console.log(err);
        return err;
    }

    return response
};

if(require.main === module){
    (async () => {
        var result = await exports.lambdaHandler(process.argv[1]);
        console.log(result);
    })();
}

I edited template.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  sam-app-2

  Sample SAM Template for sam-app-2

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x
      Layers:
        - !Ref DepLayer
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

  DepLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
        LayerName: sam-app-dependencies-2
        Description: Dependencies for sam app [temp-units-conv]
        ContentUri: ./dependencies/
        CompatibleRuntimes:
          - nodejs12.x
        LicenseInfo: 'MIT'
        RetentionPolicy: Retain
Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

Running it straight from the command line works:

sam-app-2> node hello-world\app.js
{
  statusCode: 200,
  'dep.x': 'It works!!',
  body: '{"message":"Hello, World!!"}'
}

Even sam deploy works! Yes, it deploys the code to the cloud and when I invoke the lambda function in the cloud, it gives the same result as above.

However, when I run sam build, it fails with:

Building resource 'HelloWorldFunction'
Running NodejsNpmBuilder:NpmPack
Running NodejsNpmBuilder:CopyNpmrc
Running NodejsNpmBuilder:CopySource
Running NodejsNpmBuilder:NpmInstall

Build Failed
Error: NodejsNpmBuilder:NpmInstall - NPM Failed: npm ERR! code ENOLOCAL
npm ERR! Could not install from "..\dependencies\nodejs" as it does not contain a package.json file.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\Brandon\AppData\Roaming\npm-cache\_logs\2020-03-04T19_34_01_873Z-debug.log

When I try to invoke the lambda locally:

sam local invoke "HelloWorldFunction" -e events/event.json
Invoking app.lambdaHandler (nodejs12.x)
DepLayer is a local Layer in the template
Building image...
Requested to skip pulling images ...

Mounting C:\Users\Brandon\source\repos\sam-app-2\hello-world as /var/task:ro,delegated inside runtime container
2020-03-03T19:34:28.824Z        undefined       ERROR   Uncaught Exception      {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'dep'\nRequire stack:\n- /var/task/app.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js","stack":["Runtime.ImportModuleError: Error: Cannot find module 'dep'","Require stack:","- /var/task/app.js","- /var/runtime/UserFunction.js","- /var/runtime/index.js","    at _loadUserApp (/var/runtime/UserFunction.js:100:13)","    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)","    at Object.<anonymous> (/var/runtime/index.js:43:30)","    at Module._compile (internal/modules/cjs/loader.js:955:30)","    at Object.Module._extensions..js (internal/modules/cjs/loader.js:991:10)","    at Module.load (internal/modules/cjs/loader.js:811:32)","    at Function.Module._load (internal/modules/cjs/loader.js:723:14)","    at Function.Module.runMain (internal/modules/cjs/loader.js:1043:10)","    at internal/main/run_main_module.js:17:11"]}
?[32mSTART RequestId: b6f39717-746d-1597-9838-3b6472ec8843 Version: $LATEST?[0m
?[32mEND RequestId: b6f39717-746d-1597-9838-3b6472ec8843?[0m
?[32mREPORT RequestId: b6f39717-746d-1597-9838-3b6472ec8843     Init Duration: 237.77 ms        Duration: 3.67 ms       Billed Duration: 100 ms Memory Size: 128 MB     Max Memory Used: 38 MB  ?[0m

{"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'dep'\nRequire stack:\n- /var/task/app.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js"}

When I try to start the API locally with sam local start-api, it fails with the same error as above.

I'm thinking that if it weren't for the relative file paths being off during the build phase, I'd be able to have my cake (debugging locally very quickly) and eat it too (run sam build, sam local start-api).

What should I do?

Answer 1

After much frustration and angst, this has been produced: https://github.com/blmille1/aws-sam-layers-template

Enjoy!