I've been working on a simple PE process loader for Linux. I think I've gotten the basics down; much of the code I use comes from binfmt_elf.c and binfmt_aout.c. My test executable is about is simple as possible (FASM format):
format PE GUI 4.0
entry main
section '.text' code readable executable
main:
mov eax, 4
add eax, 5
ret
This program (math1) is compiled 32-bit, and the VM I'm running in (Xubuntu 12.04) is also 32-bit. I compiled the loader as a kernel module and installed it with insmod. The loader appears to work so far, and I've checked for error codes at every possible step. All it does is mmap the code section at the start address 0x401000 and call start_thread() with that address. If I type something like ./math1.exe at the command line, my loader is indeed invoked. So, if all goes according to plan, I should get the return value 9 every time. Instead of executing, however, math1 segfaults every time, so I opened it up in gdb to see what was happening.
Before execution, everything looks normal. If I dump at the start address, I see exactly what I should (I can confirm that this is the machine code for the above program):
(gdb) x/9xb 0x401000
0x401000: 0xb8 0x04 0x00 0x00 0x00 0x83 0xc0 0x05
0x401008: 0xc3
(gdb) run
Starting program: /media/sf_Sandbox/math1.exe
Program received signal SIGSEGV, Segmentation fault.
0x00401002 in ?? ()
After the segfault, dumping at the same address, the memory has totally changed, and, from the register dump, it doesn't appear that the first instruction is ever executed:
(gdb) x/9xb 0x401000
0x401000: 0x4d 0x5a 0x80 0x00 0x01 0x00 0x00 0x00
0x401008: 0x04
(gdb) info all-registers
eax 0x0 0
ecx 0x81394e8 135501032
edx 0x64656d2f 1684368687
ebx 0x8139548 135501128
esp 0xbfffe5a0 0xbfffe5a0
ebp 0xffffffff 0xffffffff
esi 0x81394e8 135501032
edi 0x2f7ff4 3112948
eip 0x401002 0x401002
eflags 0x210296 [ PF AF SF IF RF ID ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x0 0
...more registers...
I want to know what would make something like this happen, and what I might be able to do to fix it. I'm suspecting that the code I use to set up the stack frame might be a little... off, but I'm not sure how to tell if that's what's causing this. I know this is a really specific question, but I'm hoping someone can give me some advice.
The error was caused by a faulty page-aligning on my part, and the mysterious hex dump contents were, in fact, the MS-DOS header of the PE file in question. I fixed it by using kernel_read and copy_to_user instead of do_mmap to avoid failure on non-page-aligned sections.