Search code examples
node.jsworker-thread

Is parallelization of network requests a good use for Node.js workers?


I'm experimenting with the Node.js worker_threads module, with the aim of 'collecting' the result of many requests to different APIs. Is this a good use case for Workers?

For example:

import { Worker } from 'worker_threads'

const API_ADDRESSES = [... maybe 20 different URIs]

const results = await Promise.allSettled(
  API_ADDRESSES.map(
    uri => new Promise(
      (resolve, reject) => {
        const worker = new Worker(... filepath.js, { workerData })
        worker.on('message', resolve)
        worker.on('error', reject)
      }
    )
  )
)

// The Worker then uses axios/node-fetch/etc to make a network request and returns data as a message

If this is NOT a good use case for workers, what would be a better approach? Also, if not a great idea, why is this NOT a good use case for workers?

Having tried this, it seems to work fine but I don't really know how to assess it from a performance perspective.

==== EDIT

The reason that I thought to try this approach instead of

await Promise.allSettled(API_ADDRESSES.map(uri => fetch(uri, {....})))

is that per result I might want to process the response before returning it (i.e. a result might be a lot of numbers that I want a correlation coefficient for).


Solution

  • Is parallelization of network requests a good use for Node.js workers?

    Not really. Node already parallelizes network requests.

    Is parallelization of CPU-heavy processing of network requests a good use for Node.js workers?

    Probably. If the processing of the network request is going to take any significant amount of time, parallelizing that processing can result in performance benefits. You can determine this with certainty by benchmarking both approaches.

    Just note that the key operator here is that you are parallelizing the cpu-heavy processing. The network request portion itself is already parallelized quite effectively.