Search code examples
javadebuggingassemblyjava-native-interfacex86

How does movzbl interact with a register value of 0xffffffffffffffff?

I am working on debugging a crash in the native C library part of our application, which is called from the Java side through JNI.

I've found this part of the crash file that Java left for me:

# JRE version: 6.0_16-b01
# Java VM: Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode linux-amd64 )
# Problematic frame:
# C  [binaryname.so+0x2760]  functionname+0x59

I've decompiled this:

[richg@SVR-LRH-ES-2A]$ gdb binaryname.so
...
(gdb) disas 0x275e 0x2768
Dump of assembler code from 0x275e to 0x2768:
0x000000000000275e <functionname+87>:  rex.RB clc
0x0000000000002760 <functionname+89>:  movzbl 0x230(%rax),%eax
0x0000000000002767 <functionname+96>:  test   %al,%al

Looking at my stack trace, again, and the register section, I can see:

RAX=0xffffffffffffffff, RBX=0x00002aab6cdf46c8, RCX=0x00002b70e0f15d73, RDX=0x000000005d5ffbe0
RSP=0x00000000463f9710, RBP=0x00000000463f9770, RSI=0x00002b70e0f27820, RDI=0x00000000463f9748
R8 =0x00002b70e0f27838, R9 =0x000000005cfa9828, R10=0x000000005cfa9478, R11=0x000000005cfa9440
R12=0x00002aab84654000, R13=0x00002aab6cdf46c8, R14=0x00000000463f9808, R15=0x00002aab84654000
RIP=0x00002aab79316760, EFL=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e

So %rax is 0xffffffffffffffff. That look suspicious to me.

I exhausted my knowledge of x86 some time ago, though. I've done some reading about movz and I understand what it does (casts from an 8 bit integer up to a 32-bit one by padding the low 24 bits with zeros), but I still have questions:

1) What's the significance of the 0x230 part in the call? I can see other uses of movzbl in the code which have different numbers in there.

2) Am I right in thinking that if the input register has a value larger than 8 bits (which %rax does, here), then this will crash with an overflow? (Which will be the root cause of my crash, if so.)

3) Why isn't %eax in the register dump Java has given me?

Solution

  • The 0x230 is an offset, 0x230(%rax) means [rax+0x230] in the normal world.

    So movzbl 0x230(%rax),%eax ends up meaning movzx eax, byte ptr [rax+0x230].

    There is no actual input register here, rax is supposed to hold an address. -1 doesn't look very valid as an address. That's probably the problem.

    eax is in the register dump of course - just look at the lower half of rax.