I know that displacement in MOD+R/M fields is maximally a signed 32-bit integer. However, I've seen some inconsistent behaviour when I pass a value that is too large for 32-bit signed (but small enough for unsigned 32-bit).
Case in point:
.intel_syntax noprefix
mov [eax + eax + 0xdeadbeef], al
mov [r10d + r10d + 0xdeadbeef], al
mov [rax + rax + 0xdeadbeef], al
mov [r10 + r10 + 0xdeadbeef], al
When I assemble (and dissasamble), I get the following results:
0: 67 88 84 00 ef be ad mov BYTE PTR [eax+eax*1-0x21524111],al
7: de
8: 67 43 88 84 12 ef be mov BYTE PTR [r10d+r10d*1-0x21524111],al
f: ad de
11: 88 04 00 mov BYTE PTR [rax+rax*1],al
14: 43 88 04 12 mov BYTE PTR [r10+r10*1],al
As it can be seen, for 32-bit registers (eax, r10d) the 32-bit displacement is taken literally (and interpreted as a signed 32-bit integer) and for 64-bit registers (rax, r10), it is discarded.
While I consider both to be reasonable outputs for the input, I do not immediately see the reason for this inconsistent handling. Is this documented somewhere?
Seems to be a bug in the intel syntax module. It results in an error message if you use at&t syntax:
mov %al, 0xdeadbeef(%rax, %rax)
Results in:
Error: 0xdeadbeef out range of signed 32bit displacement
Update: The bug seems to be because the baseindex flag is set after i386_finalize_displacement is called which only detects the error if it is set. A simple fix is to move the flag set earlier. I don't see any obvious problem with that, and at least fixes this issue:
--- tc-i386-intel.c.orig 2012-01-16 04:06:06.000000000 +0100
+++ tc-i386-intel.c 2014-06-14 16:13:31.238740524 +0200
@@ -835,6 +835,9 @@
memcpy (expP, &exp, sizeof(exp));
resolve_expression (expP);
+ if (intel_state.base || intel_state.index)
+ i.types[this_operand].bitfield.baseindex = 1;
+
if (expP->X_op != O_constant
|| expP->X_add_number
|| (!intel_state.base
@@ -882,9 +885,6 @@
return 0;
}
- if (intel_state.base || intel_state.index)
- i.types[this_operand].bitfield.baseindex = 1;
-
if (intel_state.seg)
{
for (;;)