I came across a pretty interesting
article that demonstrated how to remove nullbyte characters from shellcode. Among the techniques used, the assembly instructions shl and shr seemed to occupy a rather important role in the code.
I realize that the assembly instructions mov $0x3b, %rax and mov $59, %rax each actually generate the machine code instructions 48 c7 c0 3b 00 00 00. So to cope for this, the author instead uses mov $0x1111113b, %rax to fill the register with the system call number, which generates instead the machine code 48 c7 c0 3b 11 11 11, which successfully removes nullbytes.
Unfortunately, the code still doesn't execute because syscall treats 3b 11 11 11 as an illegal instruction, or this causes the code to seg fault. So what the author then did was shift %rax back and forth 56 bytes with the commands
shl $0x38, %rax
shr $0x38, %rax
After this shift, the code executes perfectly. What I want to know is how the shift instructions fixes the 48 c7 c0 3b 11 11 11 issue, and somehow makes %rax proper and syscall'able. I know that the shl/shr shifts bits left and right, meaning that shifting left moves the bits up into higher bits, and shifting right makes them lower again, because binary is read right to left. But how does this at all change the code and make it executable? Doesn't shifting back and forth essentially change nothing, putting the shifted bits exactly back where they were in the beginning?
My only theory is that shifting bits away leaves behind zeros. But I still don't see how shifting %rax forward and then back fixes the solution, because wouldn't it bring back the 11 11 11 section anyway?
Anyways, I thought this was interesting as I had never seen the shift operands before today. Thanks in advance.
Shifting is a lossy operation - if bits are shifted outside of the register, they just disappear. (Sometimes one of them is stored in a carry flag, but that's not important here.) See http://en.wikibooks.org/wiki/X86_Assembly/Shift_and_Rotate#Logical_Shift_Instructions .
The shift left (shl) operation does this:
0x000000001111113b << 0x38 = 0x3b00000000000000
The 0x111111 part would have occupied bit 64, 65, 66 etc., but %rax is a 64-bit register, so those bits vanish. Then, the logical shift right (shr) operation does this:
0x3b00000000000000 >> 0x38 = 0x000000000000003b
Giving you the number that you want. And that's all there is to it.