Write to excel on http post request in a non-blocking way

Question

I have an endpoint on my web server that handles post requests having a body containing a query for a mongodb database. The result of the query is then written to an excel file using exceljs one document at a time. It works perfectly fine responding with the file name as soon as it's ready when the query yields a reasonable amount of documents. The problem arises when the query results to 200k+ documents from multiple requests. What happens is that where usually it will take around 1 minute to finish writing and responding to a single client requesting 500k+ documents, it takes several more minutes to finish writing and responding to multiple requests with queries for the same amount of documents or even less.

The endpoint is pretty much this:

app.post('/queryToExcel', async (req, res) => {
    let cursor = mongodb.collection('collection').find(req.body);
    let filename = `./files/excel${Date.now()}`;
    let workbook = new Excel.stream.xlsx.WorkbookWriter({ filename: filename });
    let worksheet = workbook.addWorksheet("Sheet 1");

    let isFirstDoc = true;
    await cursor.forEach(document => {
        if (isFirstDoc) {
            worksheet.columns = getColumnNames(document);
            isFirstDoc = false;
        }
        worksheet.addRow(row).commit();
    });
    
    await workbook.commit();

    res.send(filename);
});

What should I do to respond within a time frame relative to the amount of documents requested with the query regardless if the web server is handling other requests for other files? Is it a thread problem or am I doing something wrong?

Answer 1

I solved by creating a forked child process on each request, like this:

Endpoint:

const { fork } = require('child_process');

app.post('/queryToExcel', async (req, res) => {
    const childProcess = fork("./writer.js");
    childProcess.send(req.body);
    childProcess.on('message', filename => {
        res.send(filename);
        childProcess.kill();
    });
});

File writer.js that is executed by the child process:

const writer = (reqBody) => {
    let cursor = mongodb.collection('collection').find(ReqBody);
    let filename = `./files/excel${Date.now()}`;
    let workbook = new Excel.stream.xlsx.WorkbookWriter({ filename: filename });
    let worksheet = workbook.addWorksheet("Sheet 1");
    let isFirstDoc = true;
    await cursor.forEach(document => {
        if (isFirstDoc) {
            worksheet.columns = getColumnNames(document);
            isFirstDoc = false;
        }
        worksheet.addRow(row).commit();
    });
    
    await workbook.commit();

    process.send(filename);
}