NodeJS fetch failed (object2 is not iterable) when uploading file via POST request

I'm trying to upload a file using native fetch in NodeJS (added in node 17.5, see https://nodejs.org/ko/blog/release/v17.5.0/).

However, I keep getting the following error -

TypeError: fetch failed
at Object.processResponse (node:internal/deps/undici/undici:5536:34)
at node:internal/deps/undici/undici:5858:42
at node:internal/process/task_queues:140:7
at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
cause: TypeError: object2 is not iterable
at action (node:internal/deps/undici/undici:1660:39)
at action.next (<anonymous>)
at Object.pull (node:internal/deps/undici/undici:1708:52)
at ensureIsPromise (node:internal/webstreams/util:172:19)
at readableStreamDefaultControllerCallPullIfNeeded
node:internal/webstreams/readablestream:1884:5)
at node:internal/webstreams/readablestream:1974:7
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

I'm using the following code to create and submit the form response -

function upload(hub_entity_id, document_path) {
  let formData = new FormData();
  formData.append("type", "Document");
  formData.append("name", "ap_test_document.pdf");
  formData.append("file", fs.createReadStream("ap_test_document.pdf"));
  formData.append("entity_object_id", hub_entity_id);

  const form_headers = {
    Authorization: auth_code,
    ...formData.getHeaders(),
  };

  console.log(
    `Uploading document ap_test_document.pdf to hub (${hub_entity_id}) `
  );
  console.log(formData);

  let raw_response = await fetch(urls.attachments, {
    method: "POST",
    headers: form_headers,
    body: formData,
  });

  console.log(raw_response);
}

Solution

Issue with form-data package:

The formData structure is not parseable by Node.js, so it throws: object2 is not iterable.

On the other hand, the sad story is formData will not be maintained anymore, and you may have noticed that two years have passed since the last version was published. So they officially announced that formData will be archived: The final nail in the coffin of formData.

Will this be the time for deprecation? form-data haven't been updated in a while and it still lacks some method that should be provided according to the spec. node-fetch@3 stopp recommended ppl using form-data due to inconsistency with spec compatible FormData and recommend that ppl use built in FormData or a spec:ed formdata polyfill that supports iterating over all fields and having Blob/File support

Solutions

1. using form-data package:

So in simple words, we need to somehow convert the form-data stream to a node.js stream. This can be done with the help of some stream methods as follows:

stream.Transform:

With the stream.Transform class from Node.js stream and passing the form-data instance, we can send the request with the built-in fetch API.

from Node.js doc:

Transform streams are Duplex streams where the output is in some way related to the input. Like all Duplex streams, Transform streams implement both the Readable and Writable interfaces.

So we can achieve it like this:

import { Transform } from 'stream';

// rest of code

const tr = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk);
  },
});
formData.pipe(tr);

const request = new Request(url, {
  method: 'POST',
  headers: form_headers,
  duplex: 'half',
  body: tr
})

let raw_response = await fetch(request);

stream.PassThrough:

Instead of returning each chunk of stream, We can simply use stream.PassThrough:

import { PassThrough } from 'stream';

// rest of code

const pt = new PassThrough()
formData.pipe(pt);

const request = new Request(url, {
  method: 'POST',
  headers: form_headers,
  duplex: 'half',
  body: pt
})

let raw_response = await fetch(request);

Important note: If you don't pass duplex: 'half', you would get this error:

duplex option is required when sending a body

2. using built-in form-data

Currently, the part of the Node.js core that handles fetch is named undici.

Luckily, you don't need to use any third-party module for handling any form-data, since undici has implemented it and is now a part of Node.js core.

Sadly, working with part of undici streaming is not easy and straightforward. However, you can still achieve it.

import { Readable } from 'stream';
// For now, it is essential for encoding the header part, or you can skip importing this module and instead implement it by yourself. 
import { FormDataEncoder } from 'form-data-encoder';

// This is a built-in FormData class, as long as you're using Node.js version 18.x and above, 
// no need to import any third-party form-data packages from NPM.
const formData = new FormData();
formData.set('file', {
  name: 'ap_test_document.pdf',
  [Symbol.toStringTag]: 'File',
  stream: () => fs.createReadStream(filePath) 
});

const encoder = new FormDataEncoder(formData)

const request = new Request(url, {
  method: 'POST',
  headers: {
    'content-type': encoder.contentType,
    Authorization: auth_code
  },
  duplex: 'half',
  body: Readable.from(encoder)
})
let raw_response = await fetch(request);

P.S: You may need to read this issue, for the part about encoding.