AWS Lambda Nodejs 20.x fails with init Status: error Error Type: Runtime.Unknown

Question

I have upgraded a previously working lambda function from node v18 to node v20.

After going through the various issues that nodejs 20 introduced (a bunch of seemingly breaking changes), I finally got a version of the function working on a local docker container.

I'm using typescript with esm module resolution, the latter of which nodejs 20.x (more or less) enforces (yes, there's an experimental tag workaround, but I can't rely on that).

I have a project with index.ts, which becomes index.js after transpilation. I have all the usual suspects as you can see below:

In order to make sure that the build works in the target environment. I set up Docker based on the image: public.ecr.aws/lambda/nodejs:20

I then use a shell script to:

1. create a local-dist directory to copy and change package.json and tsconfig.json to remove references to a "dist" directory
2. in the Dockerfile: copy both of these up to /var/task (working dir)
3. in the Dockerfile: npm install
4. in the Dockerfile: add a src directory and copy up the local src to the working dir
5. in the Dockerfile: npm run build

The container has the command index.handler, so it should run as if its a Lambda in its native environment

Once that's verified (as in it's not throwing errors relating to my code):

I then pull out: all the relevant files with:

  rm -rf dist
  mkdir dist
  docker cp "$CONTAINER_ID:/var/task/index.js" "dist/index.js"
  docker cp "$CONTAINER_ID:/var/task/index.d.ts" "dist/index.d.ts"
  docker cp "$CONTAINER_ID:/var/task/index.js.map" "dist/index.js.map"
  docker cp "$CONTAINER_ID:/var/task/package.json" "dist/package.json"

  mkdir dist/delegates
  docker cp "$CONTAINER_ID:/var/task/delegates/" "dist/"

  mkdir dist/node_modules
  docker cp "$CONTAINER_ID:/var/task/node_modules/" "dist/"

The zip em up and send them off to aws lambda with:

  pushd dist
    zip -r $LAMBDA_FUNCTION_NAME.zip ./
    aws s3 cp ./$LAMBDA_FUNCTION_NAME.zip $LAMBDA_FUNCTION_BUCKET_URL
  popd

Now all of this apparently works because in the AWS console I can see all my code:

Where locally on seemingly the same environment my code and AWS lambda's base execution scripts worked, serving up a lambda function, in the real Lambda function I get the incredibly obtuse, vague error:

2025-01-11T21:12:47.481Z    undefined   ERROR   Uncaught Exception  
{
    "errorType": "Error",
    "errorMessage": "[object Object]",
    "stack": [
        "Error: [object Object]",
        "    at Object.intoError (file:///var/runtime/index.mjs:46:16)",
        "    at Object.textErrorLogger [as logError] (file:///var/runtime/index.mjs:684:56)",
        "    at process.<anonymous> (file:///var/runtime/index.mjs:1272:32)",
        "    at process.emit (node:events:519:28)",
        "    at process._fatalException (node:internal/process/execution:188:25)",
        "    at asyncRunEntryPointWithESMLoader (node:internal/modules/run_main:129:5)"
    ]
}

With the accompanying statement: Phase: init Status: error Error Type: Runtime.Unknown

on the lambda (and in the Docker version of the lambda) the package.json is as follows:

{
  "name": "backend",
  "version": "1.0.0",
  "description": "Served by AWS Infrastructure as outlined in [infrastructure](../infrastructure/README.md)",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "build": "tsc",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "Mike Coxon (ZGF)",
  "license": "ISC",
  "devDependencies": {
    "@types/jest": "^29.5.14",
    "jest": "^29.7.0",
    "ts-jest": "^29.2.5",
    "typescript": "^5.7.2"
  },
  "dependencies": {
    "@aws-sdk/client-api-gateway": "^3.699.0",
    "@aws-sdk/client-dynamodb": "^3.699.0",
    "@aws-sdk/client-lambda": "^3.699.0",
    "@aws-sdk/client-s3": "^3.701.0",
    "@aws-sdk/client-ses": "^3.699.0",
    "@aws-sdk/lib-dynamodb": "^3.699.0",
    "@aws-sdk/types": "^3.696.0",
    "@aws-sdk/util-dynamodb": "^3.699.0",
    "@mikeycoxon/aws-lambda-event-http-router": "^0.1.0",
    "@paypal/paypal-js": "^8.1.2",
    "@types/aws-lambda": "^8.10.146",
    "@types/node": "^22.10.1",
    "@types/uuid": "^10.0.0",
    "aws-jwt-verify": "^4.0.1",
    "http-status-codes": "^2.3.0",
    "uuid": "^11.0.3"
  }
}

Does anyone have any idea of what's going on here? This stuff worked on node 18.

Or at the very least, what can I try to get any more information out of AWS's lambda execution environment?

Thanks in advance.

EDIT

I have gone back and had a look at a previous working implementation, and realised that I was previously transpiling to common JS (when using node 18). So that may be significant.

EDIT 2

OK I have a clue to what's going on, but based on AWS doco, I don't know why its happening.

In essence despite the Environment Variables being defined...

process.env.<ANYTHING> is always undefined This was not the case under Node 18.x

It turns out that from Node 20.6 onward native support for ENV variables was enabled (doing away with the need for dotenv and equivalent).

The consequence seems to be that configuring the lambda with external env variables no longer works!

maybe Node 20.6 broke lambdas process.env resolution?

So I decided to supply my own .env file as the node doco suggests:

and guess what! - it goes a little bit further

INIT_REPORT Init Duration: 3013.19 ms Phase: invoke Status: timeout

Why the time out? I mean, it's not a very big project. It might be that support for environment variables is a bit theoretical at this stage? unless something else is causing the timeout.

Boy, am i thinking... going back to Node 18x :(

EDIT 3

Increased the timeout to 10 seconds in the console, and the function is still timing out on initialisation after 3 seconds.

Far out. I need som AWS expert to tell me what's going on.

Answer 1

SOLVED

If you look at the first of the two lamda function console images, you can see the handler declaration written as:

exports.handler = async (event: APIGatewayProxyEvent) => {
...
}

this is of course, a commonjs declaration, which should have thrown a fatal exception around "no exported module" (in the context of ESM). But AWS Lambda's UserFunction.mjs seems to bury that and launch a broken instance, which responds to requests, but is not attached or referenceable in any way (explaining why changing the timeout value had no effect).

I changed the above to:

export const handler: Handler = async (event: APIGatewayProxyEvent) => {
...
}

and voilà! everything was suddenly just peachy.

In the process of trying to fix this, I hard-coded the environment variables into the application, so I haven't gone back to test if any of that works as advertised. After two-and-a-half days of tearing my hair out, I just can't be bothered.

Note to any AWS Node Lambda developers out there: If you can't find an exported user function, don't bury the exception, just throw it. It's never going to work anyway, and all you're doing is hiding the true cause of the issue from the developer, behind one of the most obtuse error messages I think I've ever encountered, namely:

Status: error Error Type: Runtime.Unknown