AWS Lambda Node Function Silently Fails

I have an AWS Lambda function served through the AWS API gateway. Everything seems to work fine for the most part. There is one function of our API that isn't working right. It is a webhook that receives incoming faxes and the emails us to let us know it came in.

We have tested everything and this works perfectly on our local machine and even on providers like linode or digital ocean. Our problem is only when deployed with AWS Lambda.

I have tried increasing memory and execution limits well beyond what is required.
I've tried with multiple email methods and libraries.
I've tried with various function formats.

The cloudwatch logs for lambda show the console.log(req.body.MediaUrl); but there is no other output on stdout or stderr. Even if I put obvious flaws into it there will be no errors. It's almost as if the function silently fails without explaining why.

I have been able to get it to work if I send 2-3 post requests all within about 1-2 seconds of another. 1 maybe 2 emails will be sent, but never all of them. If a single post request is sent as is normal usage you never get an email.

// Define a handler for when the fax is initially sent
const nodemailer = require('nodemailer');
const aws = require('aws-sdk');

// Start Email Logic
// AWS access keys are built into Lambda, modify permissions of the executor role
aws.config.update({region: 'us-west-2'});

let transporter = nodemailer.createTransport({
  SES: new aws.SES({
    apiVersion: '2010-12-01',
  }),
 });

exports.received = function(req, res) {

  transporter.sendMail({
  from: '"Fax" <fax@domain.com>',
  to: process.env.FAX_ADDRESS,
  subject: 'A new fax from ' + req.body.From,
  text: 'You can view the fax at this url:\n\n' + req.body.MediaUrl,
}, (err, info) => {
  if (err) {
    console.log(err);
  } else {
    console.log(info.envelope);
    console.log(info.messageId);
    console.log('Email Sent');
  }
});

// log the URL of the PDF received in the fax just in case email fails
console.log(req.body.MediaUrl);

  res.status(200);
  res.send();
};

BTW the above code has another file with all the proper express items to properly serve this. Again, it works perfectly on local, other servers and even sometimes when flooding lambda; it never consistently works on Lambda though.

My guess is either I'm an idiot that needs to sleep or I'm missing something about Lambda here. Can anyone shed some light on what I'm missing?

EDIT :

Thanks to Michael and Hen things fell into place. When working with Lambda the lambda function will shut down the moment it gets through the main process logic. It won't wait for pending asynchronous items to complete. The following change fixed my issue when using Express with Lambda.

Notice that I've moved the mail functions to the end of the response and then moved the response status and send to inside of the mail function. This is a one of many ways this can be solved but is sweet and simple for our needs.

exports.received = function(req, res) {

  // log the URL of the PDF received in the fax just in case email fails
  console.log(req.body.MediaUrl);

  transporter.sendMail({
  from: '"Fax" <fax@domain.com>',
  to: process.env.FAX_ADDRESS,
  subject: 'A new fax from ' + req.body.From,
  text: 'You can view the fax at this url:\n\n' + req.body.MediaUrl,
}, (err, info) => {
  if (err) {
    console.log(err);
    res.status(500);
    res.send();
  } else {
    console.log(info.envelope);
    console.log(info.messageId);
    console.log('Email Sent');
    res.status(200);
    res.send();
  }
});
};

Solution

It isn't silently failing, it's just going into suspended animation. Trying it again in short succession is actually allowing the earlier attempts to have enough runtime to finish.

The problem seems to be here:

res.status(200);
res.send();

These should be inside the callback from sendMail(). You shouldn't return success until after success, but you're returning it before that, since sendMail() is asynchronous.

You can't expect Lambda to keep doing things for you after you've told it you're done. Lambda is request/response.¹ Start function, do stuff, return response, stop. The process is frozen, and runtime billing stops. If there are things still on the event loop, then they just hang, depending on the value of context.callbackWaitsForEmptyEventLoop, which I doubt should/could be safely changed with express.

If the next function invocation reuses the same container (a decision made by the infrastructure), then things you left running are still running when the container is thawed, so they may subsequently finish if the invocations are close enough together in time that sockets haven't timed out or any other potential wall-clock-time-related failures haven't occurred.

Only one invocation runs in any one container at any one time, but here you're leaving things running, so work from a previous invocation is actually finishing inside a later one.

If you duplicate the console.log(req.body.MediaUrl); line both inside the callback and leaving it where it is now, assuming that value is different for each request, you should actually capture some evidence to confirm what I'm asserting, here.

¹Lambda is request/response. Lambda functions themselves can also be invoked asynchronously as "event" invocations, so that the "response" to the external caller is only an indication that the Lambda infrastructure has agreed to execute the function for you, in cases where you don't want or need to wait for the actual response, but that isn't relevant in this context. Lambda functions are still request/response, even though under this alternate model the response from the function is discarded, and the invocation is retried twice if an exception is thrown.