The assignment is to interpret some assembly code to find a phrase that will run without calling "explode bomb". I'm using gdb to disassemble the current phase. I'm stuck and any advice on whether I'm moving in the right direction would be very appreciated.
0x0000000000400fb5 <+0>: sub $0x18,%rsp
0x0000000000400fb9 <+4>: lea 0x8(%rsp),%r8
0x0000000000400fbe <+9>: lea 0xf(%rsp),%rcx
0x0000000000400fc3 <+14>: lea 0x4(%rsp),%rdx
I believe what's happening is that we're looking at the phrase in %rsp (what the base pointer is pointing to). I think it's four things saved in memory, at (%rsp), 0x4(%rsp), 0x8(%rsp), and 0xf(%rsp), which I think makes it three integers and something else, but I'm not sure what.
0x0000000000400fc8 <+19>: mov $0x4028f6,%esi
The value at address $0x4028f6 is “%d %c %d”. I know it's being moved to register %esi, but I'm not sure how this is being used.
0x0000000000400fcd <+24>: mov $0x0,%eax
0x0000000000400fd2 <+29>: callq 0x400c80 <__isoc99_sscanf@plt>
Not sure what's happening here. I looked at 0x400c80 and found the following:
0x0000000000400c80 <+0>: jmpq *0x20340a(%rip)
0x0000000000400c86 <+6>: pushq $0x12
0x0000000000400c8b <+11>: jmpq 0x400b50
I don't see %rip being used as a register, so I'm not quite sure what to do or where to look. I think it's putting 0x12 in %eax, but then I'm not sure where it's jumping, since 0x400b50 isn't a valid address. Anyway, back to the original function:
0x0000000000400fd7 <+34>: cmp $0x2,%eax
0x0000000000400fda <+37>: jg 0x400fe1 <phase_3+44>
0x0000000000400fdc <+39>: callq 0x401719 <explode_bomb>
I think we just pushed 0x18 on %eax, so it'll definitely be greater than 0x2 and the program will skip over "explode bomb". I just want to know where I'm messing up until that point.
Unfortunately you seem to be missing crucial knowledge. You should revisit any material you might have otherwise you won't have much luck with the rest of the assignment which will actually involve some less trivial code.
That said, the block you have quoted so far only does the following:
int x, y;
char c;
if (sscanf(foo, "%d %c %d", &x, &c, &y) <= 2) explode_bomb();
The first block allocates the 3 local variables, and loads the address of each into the appropriate register for the respective sscanf argument. Note that x86-64 calling convention uses the registers rdi, rsi, rcx, rdx, r8 and r9 for the first six arguments (assuming they are appropriate) and values are returned in rax.
You don't need to look at 0x400c80, you only find the PLT entry that eventually jumps to sscanf itself. We already know it's sscanf from the symbol name.