Can I output multiple files to STDOUT and process them individually?

I'm using ImageMagick's convert program to split a pdf file into individual images. It allows you to write the output to STDOUT by specifying '-' as the output file.

I'd like to process all these images and then re-merge them. Unfortunately, when I pipe the output into a new process, it doesn't recognize the stream as individual image files. Writing all the images to the disk and then reading them with a different program would obviously be a waste of time, so is there a way to write multiple files to STDOUT pipe such that each one is the STDIN of it's another image-editing instance? Can the output of these image-editing instances be merged into the input of a final merging process?

Solution

You can't really send multiple files to stdout in any standardized way that tools generally recognize. stdout and stdin are streams, so there's no way to tell if any given piece is from one file or another.

Some tools have standard ways of delimiting records on stdout and stdin, like newlines or null characters, but that doesn't work for arbitrary data, like most image formats which may contain either.

You have a few options:

Just use multiple files. It probably won't be as slow as you think; if the files aren't too big, everything may wind up happening in cache and never actually hitting the disk.
Use a format that supports multiple frames or layers. As long as the tools that will be editing understand this format, you should be fine. Essentially, this format becomes the delimiter between the different files. One format that is relatively standard and which supports multiple pages/layers is TIFF:
```
convert my.pdf tiff:- | convert - -flip - | convert - my.flipped.pdf
```
If you are using ImageMagick for editing the files, don't split it up, edit each page, and recombine, just do it all in one pass. For example:
```
convert my.pdf -flip my.flipped.pdf
```
will flip the pages in your PDF, giving you a new PDF, all in one go.