Node.js Fetch() PDF File Buffer In Memory for Nodemailer Attachment

I have a nightly process that sends scheduled reports by calling an Endpoint I created that returns a live report in PDF Format.

The problem I ran into was getting the binary file returned from the Endpoint into a Buffer that Nodemailer can use to attach the PDF report.

I am just using the Promise returned from fetch() and arrayBuffer(). I am NOT using Async Await.

The below Code Snippet using arrayBuffer() and Buffer.from() works, but I am wondering if there is a more efficient way to process this, especially when working with large PDF Files.

I prefer to work in Memory rather than writing to disk. I have allocated a lot of Memory in Express and am NOT seeing any Memory issues yet.

const sendPDFReport = (reportScheduleId, cb) => {

let scheduleObj;
//*** Get Report Schedule Data from MongoDB and Assign to scheduleObj

let emailAddresses = [];
//*** Push Recipient Emails into Email Array

//*** Do Work Like Build Fetch URL and Connection Properties Object ie., 

let fetchBody={};
//*** Build JSON object of POST Params and assign to fetchBody

let fetchURL = process.env.APPSERVER_URL+some_report_path;

let config = {
    method: 'POST',
    headers: {'Content-Type': 'application/json'},
    body: JSON.stringify(fetchBody),
    responseType: 'blob'
};

let transporter = nodemailer.createTransport({
    host: 'mail.whatever.com',
    service: "Outlook365",
    secure: true,
    port: 465,
    auth: {
        user: process.env.NO_REPLY,
        pass: process.env.NO_REPLY_PW
    },
    tls: {
        ciphers: 'SSLv3',
        rejectUnauthorized: false
    }
});


   fetch(fetchURL, config)
    .then(response => {

        if (response.ok) {

        //*** PDF Binary Response Buffering for Nodemailer
            
        response.arrayBuffer()
           .then(resBufferAr => {
             const pdfBuffer = Buffer.from(resBufferAr);
             
             // **** Build Email mailOptions for Nodemailer Transporter Object ***

             let mailOptions = {
                 from: process.env.NO_REPLY,
                 to: emailAddresses.join([separator = ',']),
                 subject: 'Scheduled Report:  ' + scheduleObj.reportType + ' ' + scheduleObj.selectedReport,
                 html: '<h4> See Attached Report </h4>';
             
              //*** Nodemailer Attachment Section

                 attachments: [{
                     filename: scheduleObj.reportType + '_' + scheduleObj.selectedReport + '_' + now + '.pdf',
                     content: pdfBuffer,
                     encoding: 'base64',
                     contentType: 'application/pdf'
                 }]
              };

             transporter.sendMail(mailOptions, function (err) {
                if (err) {
                   console.log('Transporter Error:  ' + err);
                   return cb(err);
                }
                return cb(null);
             }
   })
   .catch((err) => {
            console.log('Problem Processing Alert Notification:  ' + err);
            return cb(err);
   })

    return cb(null);

}

Any comments or suggestions would be great.

Solution

Yes, there is a more efficient way: Streams

The short version is, streams are built-in data structures within Node which allow you to operate on chunks of incoming data rather than waiting for the entirety of the data to be loaded into memory before processing it.

node-fetch supports streams as response payloads and nodemailer supports streams as attachments, so you can effectively pipe the body of your fetch response into nodemailer. Theoretically then your machine never actually has the whole PDF in memory at any one time, and instead only ever has pieces of the PDF which it'll just pass from one side network to the other, as they come in (though there are reasons why this sometimes doesn't hold up).

I haven't fully tested this, but this should at least be close to a working solution involving streams between node-fetch and nodemailer:

if (response.ok) {
    let readableStream = response.body;

    let mailOptions = {
        from: process.env.NO_REPLY,
        to: emailAddresses.join(','),
        subject: 'Scheduled Report:  ' + scheduleObj.reportType + ' ' + scheduleObj.selectedReport,
        html: '<h4> See Attached Report </h4>',

        attachments: [{
            filename: scheduleObj.reportType + '_' + scheduleObj.selectedReport + '_' + now + '.pdf',
            content: readableStream,
            encoding: 'base64',
            contentType: 'application/pdf'
        }]
    };

   // ...the rest of your code

UPDATE: Though I didn't use node-fetch, I did test memory usage in reading a 3MB file from the disk and emailing it to myself via nodemailer. After reading the entire file into memory and then passing it off to nodemailer, the resident set size (rss) was ~48MB, and the external C++ heap was ~11MB. When giving nodemailer a read stream of the file instead, the rss was ~37MB, and the external heap was ~4MB. So it's definitely more efficient.