Search code examples
javascriptnode.jsioipc

How to write string data to a stream in chunks, not a lump


I have a node.js script, which must accept utf8 data on STDIN and output other utf8 data on STDOUT It's intended to accept a TAP string(s) from the tape testing framework runner through a shell pipe and slightly modify it:

#!/usr/bin/env node
'use strict';

process.stdin.setDefaultEncoding('utf8');
process.stdout.setDefaultEncoding('utf8');
process.stdin.on('readable', () => {
  let chunk;
  while ((chunk = process.stdin.read()) !== null) {
    process.stdout.write('chunk: ' + chunk);
  }
});

process.stdin.on('end', () => {
  process.stdout.write('end\n');
});

When I run it with a tape test npm run test-unit test/unit/main.test.js | ./format.js I get (further I'll refer to it as the "live output"):

chunk: 
> [email protected] test-unit /path/dummy/zzz
> tape "test/unit/main.test.js"

chunk: TAP version 13
chunk: # test 1
chunk: # comment 1
chunk: ok 1 should be equal
chunk: 
1..1
chunk: # tests 1
# pass  1

# ok
chunk: 
end

So you can see, the data, sort of, were written 8 times, in 8 chunks.

My question is, how can I simulate this behavior in a test?

In the test I spawned another process with spawn('./format.js') but how can I write to that process' STDIN so that it'd take 8 times to read the same data, as from the tap reporter.

I split the original tap output (without the "chunk" marks) into an array of strings (equivalent to the chunks from the live output) and tried to write each string in a separate write call to the spawned.stdin then test the format.js's output:

let spawned = spawn('./format.js');
let chunksOriginal = [
  // chunks here
];


spawned.stdout.on('readable', () => {
  console.log(spawned.stdout.read());
});
chunksOriginal.forEach((chunk, i) => {
  spawned.stdin.write(chunk);
});

I expected console.log to spew the same string as the live output, or at least to be only the first chunk from it but I got:

chunk: 
> [email protected] test-unit /path/dummy/zzz
> tape "test/unit/main.test.js"

TAP version 13
# test 1
# comment 1
ok 1 should be equal

1..1
# tests 1
# pass  1

# ok

end

As you can see the data were read by the format.js just once, with one chunk mark above. I tried to emit the "readable" event in the forEach loop before the write call, that didn't help either:

chunksOriginal.forEach((chunk, i) => {
  spawned.stdin.emit('readable');
  spawned.stdin.write(chunk);
});

I suspect that the problem is that I don't understand conceptually IPC in node.js and bash. Data in the "live output", probably, is written via a so-called node.js Buffer.

I'm not asking for an exact solution to the problem, though I'd be greatly thankful if someone provided it.

Just a hint on a conceptual level, where should I look, in what docs, to solve this would be appreaciated


Solution

  • Also I tried to use setTimeout, to make the write process asynchronous, though I forgot to mention this in the question, but it didn't help then either:

    chunksOriginal.forEach((chunk, i) => {
      setTimeout(() => {
        spawned.stdin.emit('readable');
        spawned.stdin.write(chunk);
      }, i);
    });
    

    Then I gave it one more try and increased the delay i * 100 ms namely, and it worked:

    chunksOriginal.forEach((chunk, i) => {
      setTimeout(() => {
        spawned.stdin.write(chunk);
      }, i * 100);
    });
    

    That helped, now the format.js receives the same amount of chunks from the dummy input, as it receives from the live one (though it all may need some tuning, config creation).

    Also it's worth to note that the emit call is unnecessary.