How to keep alive a connection from angular to nodejs

Question

I'm making a nodejs(10.16.0) and angular(8.0.3) app.

One of the modules of this app is intended to retrieve data from a database by executing a pretty huge query (read-only purposes, 80 lines with some inner joins and several conditions), at the end, depending on the sent parameters, this process can retrieve from 0 to more than 600,000 rows (or even more), so query execution can take from 1 second to even 4 minutes.

Process is the next:

1. User insert (from front end) several parameters.
2. User clicks on generate button.
3. Angular builds a JSON with the data needed to form the query and sends it to Nodejs server.
4. Nodejs receives JSON and forms this query (here the time starts to count, the whole process can take 0 - 240 seconds).
5. Nodejs asks DB2 server for the result of the query (This is the sub-process that takes longer, even several minutes)
6. Nodejs gets the result.
7. Nodejs sanitizes result (0 - 5 seconds depending on the array length).
8. Nodejs returns the sanitized list to angular.
9. Angular creates a file with the sanitized list (another 0 - 5 seconds).
10. Download file.

My angular request (function called after click on send button):

  // User input is the JSON that contains parameters
  getReportData(userInput: JSON): Observable<MyInterface> {
    return this.httpClient.post<MyInterface>(`${this.MyURL}/generateReport`, { userInput })
      .pipe(
        catchError(e => throwError(new Error("No Cost Types found" + e.message))));
  }

My nodejs endpoint response:

router.post('/generateReport', async (req, res, next) => {
    try {
        // ** Here it is when the time starts to count**
        const result = await dataRetriever.getData(req.body.userInput);
        if (result instanceof Error) next(createError(result, 400));
        else res.send(result);
    } catch (error) {
        next(createError(error, 400));
    }
});

Data retrieve function:

async function getData(userInput) {
    const query = queryGenerator.getCQRDataRetrieveQuery(userInput);
    // This await is the process that can take several minutes
    const data = await queryExecutor.executeStatement(query);
    if (data.length >= 1) {
        const result = sanitizeDBResults(data);
        return result;
    } else {
        // In case no rows were found
        return [];
    }
}

The problem is that when angular waits for the response after clicking on 'generate' button it shuts down the connection after 60 seconds of waiting (Which is less time than the already seen examples).

This is the message shown on the chrome console after 60 seconds of waiting:

zone.js:3372 POST https://*myURL*/generateReport 504 (Gateway Time-out)

I have heard about a header called 'keep alive' or implementing web sockets to 'heartbeat' from nodejs to angular so front end knows connection is still alive, but are these the correct approaches? What another solution could be simple and effective?

So what I need to implement is a way to keep Angular hearing for a response for more than 60 seconds, query results can take 4 minutes or even more if needed.

Any idea about this?

Answer 1

I particularly don't think the part of assembling the file with JS is necessary. You could simply add the Content-Disposition header on top of Content-Type and download it directly. Example:

Content-Disposition: attachment; filename = "report.txt"

But if the problem remains about the timeout, you can specify this to Express:

Globally:

const server = app.listen();
server.setTimeout(600000);

Or just for specific route:

app.post('/xxx', (req, res) => {
    req.setTimeout(600000);
});

And as it is a pretty big file, I recommend you take a look at: Send data in chunks with nodejs