Search code examples
bashunixshpipelinefile-descriptor

How to use a different file-descriptor in a shell pipeline?

I'm dealing with a script, that invokes a noisy (lots of diagnostics on both stdout and stderr) program first, and then processes its output with other tools.

The program's verbosity makes it impossible to simply send its stdout to pipeline, so currently we use a temporary file -- a practice I'd like to end.

Instead of /tmp/foo, we can ask the program to write the data to /dev/fd/N -- and it will, no problem (it does not need to seek the file, for example).

What noise it currently sends to stdout and stderr, can continue going there -- the operators are used to seeing it, and will be alarmed, if it disappears...

But how do I arrange for the descriptor N to exist and be sent into the next program's stdin?

noisy -o /dev/fd/N ?????| filter -i /dev/stdin

If this requires bash, so be it, but I'd prefer a solution suitable for the entire sh-family, of course.

Solution

  • If I understand your problem correctly, you've got a program that writes noise to standard output and standard error, and writes useful data to a file specified with a -o option. You want standard output and standard error to be left as they are, but pipe the useful data into a filter program instead of writing it to file.

    The easiest way to do that with Bash is to use process substitution (see ProcessSubstitution - Greg's Wiki):

    noisy -o >(filter -i /dev/stdin)

    Note that process substitution is not available in some sh-family shells, it is not available with Bash on some (uncommon) platforms, and there is no way to get the exit status of a process created with process substitution with Bash before version 4.4.

    Another possible way to do what (I think) you want is:

    exec 3>&1
{ exec 4>&1; noisy -o /dev/fd/4 >&3 ; } | filter -i /dev/stdin
    • exec 3>&1 makes file descriptor 3 refer to the "real" standard output.
    • exec 4>&1 (since it is run in a process that is the first stage of a pipeline) makes file descriptor 4 refer to the input to the next stage in the pipeline.
    • noisy ... >&3 forces the standard output of noisy to go to the "real" standard output.
    • Writing to /dev/fd/4 (on Linux at least) writes to the next stage in the pipeline.

    I've only tested it with Bash, but I think it should work with other sh-family shells.