gulp-foreach not looping over all files

I have a very basic usage of gulp-foreach that's failing so badly I imagine I must be doing something wrong; I'm just (for now) trying to print all files found in gulp.src. When doing this with gulp-foreach only a very, very small sample of the files are ever printed—about 15 out of many hundreds. When using gulp-print things work as expected.

const gulp = require('gulp'),
      foreach = require('gulp-foreach'),
      gulpPrint = require('gulp-print');

gulp.src([`../../**/*.js`].concat([]))
    .pipe(gulpPrint(path => {
        console.log(path);
    }))
    .pipe(foreach((stream, file) => {
        console.log(file.path);
        return stream;
    }))

The code with the foreach is triggered about 15 times, interleaved with the gulpPrint, followed by a flood of gulpPrint statements alone. If I comment out the gulpPrint part, foreach prints about 15 times and then exits, and gulpPrint alone, as expected, floods the console with every file found, as it should.

I can just use gulp-print, but I'd like to know what I'm doing wrong with gulp-foreach.

Solution

First of all: gulp-foreach isn't really the right plugin for your use case.

gulp-foreach allows you to create a substream for each file and emit the results of the substream into the original stream. That means you can emit multiple files for each input file (see this example). In this regard it is very similar to the higher-order function flatmap in functional programming. Which is why the author of gulp-foreach has recently renamed the plugin to gulp-flatmap.

All you are interested in is to access each file in the stream and print its path, which makes this a case of the higher-order function map. The right plugins for this are map-stream or gulp-tap (which is what the author of gulp-foreach used to recommend).

Now to your actual question.

As already mentioned gulp-foreach emits all the files from the created substream to the original stream. However you are doing nothing with the files that are written to the original stream. Node.js object streams by default have a highWaterMark of 16, so as soon as gulp-foreach has written 16 files to the original stream the buffer is full and gulp-foreach blocks.

In order to fix this you need to consume the files that gulp-foreach writes to the stream. You could simply use a dummy .on('data') handler for this:

gulp.src([`../../**/*.js`].concat([]))
  .pipe(gulpPrint(path => {
    console.log(path);
  }))
  .pipe(foreach((stream, file) => {
    console.log(file.path);
    return stream;
  }))
  .on('data', (data) => { });

However the usual way to do this in gulp is to let gulp consume the stream instead by simply returning the stream from the task it is created in:

gulp.task('default', function() {
  return gulp.src([`../../**/*.js`].concat([]))
    .pipe(gulpPrint(path => {
       console.log(path);
    }))
    .pipe(foreach((stream, file) => {
       console.log(file.path);
       return stream;
    }));
});