Search code examples
assemblyx86-64attmachine-codeinstruction-encoding

What is REX prefix in Instruction Encoding?

In instruction encoding we have an optional field REX prefix which is split like this:

Field contains
0100 – fixed constant value 
W – 1 when using 64-bit data
R – expands the Reg field to 4 bit 
X – expands Index field to 4 bit
B – expands the R/M field or Base to 4 bit

I asked my professor what "when using 64-bit data" really means and every time I get different answer:

  1. When using the new registers added to x64 like r9, r10, r9d etc...

  2. When using 64 - size register like rax, rbx, r9 etc...

  3. When the instruction uses (read/write) 64 bit data from memory.

I'm really confused, what's the correct answer (it could be none of these as I don't trust my professor anymore).

Solution

  • All three cases need a REX prefix:

    • #2 and #3 need the W bit set in the REX because they're "using 64-bit data" (operand-size)
    • #1 needs one of the other bits set, depending on which register(s) are "high" registers

    For instance, addq %rax, %rbx, addq %rax, long_var, and addq $123, long_var all need the REX prefix with W bit set, since they all have a 64-bit operand size. addq %rax, %r9 needs both W and B because of the use of the "new" register.

    However, addl %eax, %r9d doesn't need the W bit, since its operand size is 32 bits. It does still does need the B bit to use the new register r9d, so the prefix byte is still needed, but not because of "using 64-bit data". (Case 1)

    It's possible to have multiple cases at once. addq (%r8, %r9, 8), %r10 would need all four bits set in the REX prefix, including W.

    An example of pure case #3 would be movq $123, 0x1000 to store an 8-byte integer 123 (sign-extended from 32-bit) to memory at absolute address 0x1000, no registers at all involved, but the 64-bit operand-size can only be set with a REX prefix with W=1. More normally you'd movq $123, symbol(%rip) to reference static storage, and that still doesn't include any general-purpose integer registers, or movq $123, 8(%rsp) where the addressing mode involves are register but neither the source nor destination actually are registers.

    See also https://wiki.osdev.org/X86-64_Instruction_Encoding#REX_prefix

    A few opcodes default to 64-bit operand-size and don't need a REX prefix; for example pushq $1 assembles to 6a 01. (And pushl is not encodeable; a REX.W=0 can't override it back to 32-bit. Only the 66 operand-size prefix can change it, to 16-bit so pushw is encodeable.) This is mostly stack operations, including call / ret and indirect calls / jumps, like ff 17 callq *(%rdi) that loads a new 64-bit RIP from memory, as well as pushing a 64-bit return address.