How to create self-decompressing executables on linux when you cannot execve a pipe

I've been working on a small hack to get the filesize of an executable down. I'm aware there exist tools that do executable compressing properly, but this is more for my own enjoyment than anything serious.

My idea is to compress the executable with gzip, then embed it in another c program, called the launcher, as an array. When the launcher runs, it sets up a piping system like so:

parent launcher -> fork 1 of launcher -> fork 2 of launcher

fork 1 turns itself into gzip, so it decompresses whatever the parent feeds it, and spits out the decompressed version to fork 2.

Here's where the hack kicks in. Fork 2 tries to exec the file "/dev/fd/n", where n is the file number of the pipe that goes from fork 1 to fork 2. In essence this means fork 2 will try to execute whatever binary gzip spits out.

However, this doesn't work (surprise surprise.) I tried stracing my sample implementation and the line that does the execv on "/dev/fd/n" returns -1 EACCES (Permission denied). However, if I open up a terminal and run ls -l /dev/fd/ I get something like:

lrwx------ 1 blackle users 64 Nov 10 05:14 0 -> /dev/pts/0
lrwx------ 1 blackle users 64 Nov 10 05:14 1 -> /dev/pts/0
lrwx------ 1 blackle users 64 Nov 10 05:14 2 -> /dev/pts/0
lr-x------ 1 blackle users 64 Nov 10 05:14 3 -> /proc/17138/fd

All of them have permissions +x for the user (me.) This means it should be executable, no? Or is this just a very strange kernel edge case that says it hasn't got the permissions, but really it can't execute because it's not a real file.

UPDATE after nearly 7 years

With the advent of linux "memfd"s, it's now possible to create self-decompressing executables on linux that don't touch the filesystem. See: https://gitlab.com/PoroCYon/vondehi

Solution

Only files which are mmap able are possible to execute. A pipe is unfortunately not mmapable in this way due to its sequential nature and limited buffer size (it may need to re-read earlier code again which would now be gone after reading it once).

You would have much more luck instead of using a pipe to create a file in a ramfs, mmap it to an area of the memory space of the parent the copy the uncompressed code into the mmap, then finally have the child exec the file in the ramfs, finally unlink the file in ramfs in the parent so it is automatically freed when the child exits.

Hope this helps, if anything is unclear please comment.