How to run shell-liked pipe tasks with limited fork number?

I have a simple program, which I want to simulate the condition that I don 't have enough fork capacity, so I limit the fork number when doing pipe tasks.

let the shell-liked pipeline job written in C++:

ls | cat | cat | cat | cat | cat | cat | cat | cat

I have the code, which run pipe() and fork():

#include <errno.h>
#include <fcntl.h>
#include <iostream>
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

const int fork_limit = 3;
int fork_counter = 0;

static void sig_chld_handler(int signo) {
  int status;
  pid_t pid;
  while ((pid = waitpid(-1, &status, WNOHANG)) > 0) {
    printf("received SIGCHLD from child process %d\n", pid);
    fork_counter -= 1;
    fprintf(stdout, "counter --, %d\n", fork_counter);
  }
}

int main(int argc, char **argv) {

  signal(SIGCHLD, sig_chld_handler);

  char **cmds[9];

  char *p1_args[] = {"ls", NULL};
  char *p2_args[] = {"cat", NULL};

  cmds[0] = p1_args;
  cmds[1] = p2_args;
  cmds[2] = p2_args;
  cmds[3] = p2_args;
  cmds[4] = p2_args;
  cmds[5] = p2_args;
  cmds[6] = p2_args;
  cmds[7] = p2_args;
  cmds[8] = p2_args;


  int pipes[16];
  pipe(pipes);     // sets up 1st pipe
  pipe(pipes + 2); // sets up 2nd pipe
  pipe(pipes + 4);
  pipe(pipes + 6);
  pipe(pipes + 8);
  pipe(pipes + 10);
  pipe(pipes + 12);
  pipe(pipes + 14);


  pid_t pid;

  for (int i = 0; i < 9; i++) {

    // === comment this part to run correctly ===
    while (fork_limit < fork_counter) {
      usleep(10000);
    }
    // ===

    pid = fork();
    if (pid == 0) {
      fprintf(stdout, "fork p%d\n", i);

      // read
      if (i != 0) {
        if (dup2(pipes[(i - 1) * 2], 0) < 0) {
          fprintf(stderr, "dup2 error\n");
          exit(EXIT_FAILURE);
        }
      }

      // write
      if (i != 8) {
        if (dup2(pipes[i * 2 + 1], 1) < 0) {
          fprintf(stderr, "dup2 error\n");
          exit(EXIT_FAILURE);
        }
      }

      for (int j = 0; j < 16; j++) {
        close(pipes[j]);
      }

      execvp(*cmds[i], cmds[i]);
    } else {
      fork_counter += 1;
      fprintf(stdout, "counter ++, %d \n", fork_counter);
    }
  }

  for (int j = 0; j < 16; j++) {
    close(pipes[j]);
  }

  waitpid(pid, NULL, 0); // wait the last one.

  std::cout << "Parent done." << std::endl;
}

the line while (fork_limit < fork_counter) is what I restrict the child number. If I remove the while block, the code runs well, but it hangs if I add this.

I suppose the previous children will die, so that fork_counter -= 1, and the new child can be forked, but the behavior is not and I cannot figure out why.

Result of without the while.

counter ++, 1 
counter ++, 2 
fork p0
fork p1
counter ++, 3 
fork p2
counter ++, 4 
counter ++, 5 
fork p3
fork p4
counter ++, 6 
fork p5
counter ++, 7 
counter ++, 8 
fork p6
fork p7
counter ++, 9 
fork p8
received SIGCHLD from child process 13316
counter --, 8
Applications
Desktop
Documents
Downloads
Library
Movies
Music
Pictures
received SIGCHLD from child process 13319
counter --, 7
received SIGCHLD from child process 13318
counter --, 6
received SIGCHLD from child process 13317
counter --, 5
received SIGCHLD from child process 13320
counter --, 4
received SIGCHLD from child process 13322
counter --, 3
received SIGCHLD from child process 13321
counter --, 2
received SIGCHLD from child process 13323
counter --, 1
received SIGCHLD from child process 13324
counter --, 0
Parent done.

Result of having the while, which means I limit the fork number.

counter ++, 1 
counter ++, 2 
fork p0
fork p1
counter ++, 3 
counter ++, 4 
fork p2
fork p3
received SIGCHLD from child process 13291
counter --, 3
counter ++, 4 
fork p4

(hang)

Solution

The main program perform (in sequence) the following:

Pre-create all the pipes
Fork children using the pipes (each children closes all inherited pipes)
Close all pipes

The problem is with the timing of 'close all pipes'. Because the main is waiting for the first children to complete (while (fork_limit < fork_counter)), before it can finish step #2.

However, the cat children (e.g., the 1st cat) can not finish until their input pipe is closed by ALL processes, including he main, which is waiting for them to finish. Effectively a deadlock.

Consider small modification to the main process, which will close the pipes to each children, as soon as the children is forked:

if ( fork() ) {
   // Children
   ...

} else {
   // Main - close pipes ASAP.
      close(pipes[(i-1)*2]) ;
      close(pipes[(i-1)*2+1]);
      fork_counter += 1;
      fprintf(stdout, "counter ++, %d \n", fork_counter);
}

Probably some modification to the pipe closing in the children is also needed.