The instruction
mov al, byte ptr [rbx + rsi*2 + 0x100]
is encoded to
8a 84 73 00 01 00 00
meaning that ModR/M is 0x84 or 10.000.100
I understand the mod (10) and the reg (000) fields, but don't know why rm is 100. Could anyone explain the rule used here?
The rm field is 100 to indicate a SIB (scale index base) operand. This operand is elaborated in the SIB byte, which if present, immediately follows the modr/m byte. In this case, the SIB byte is 0x73 or 01.110.011, indicating a scale (01) of 2, rsi (110) as the index register and rbx (011) as the base register.
Refer to the Intel manuals for details.