[x64]
is there some generic algorithm using which I can determine relevant registers for the rex byte? I'm specifically interested in the calculation of REX for instructions with the VEX prefix (so that I can determine ~R ~X ~B)?
I've tried using the first and last register operand as base and rx, but that only seems to work for stuff like andn, but fails on bextr.
for example -
andn ecx, r15d, edx
bextr ecx, r15d, edx
respectively encode to:
c4 e2 00 f2 ca
c4 c2 68 f7 cf
I'm confused about the second VEX byte, which is formed like this
~R [X_______]
~X [_X______]
~B [__X_____]
map_select [___XXXXX]
e2: 1110 0010 (R = 0, X = 0, B = 0)
c2: 1100 0010 (R = 0, X = 0, B = 1)
why does bextr encode with B = 1 while andn doesn't?
Look at the field Op/En of instructions ANDN and BEXTR.
ANDN r32a, r32b, r/m32 Bitwise AND of inverted r32b with r/m32, store result in r32a
RVM says that the middle operand r32b is encoded as vvvv field of VEX3, while in
BEXTR r32a, r/m32, r32b Contiguous bitwise extract from r/m32 using r32b as control; store result in r32a.
is the middle operand encoded as field r/m of ModRM (RMV).
If you used one of the low seven registers instead of r15d, your problem would vanish and all bits R,X,B would be 0:
|C4 E2 78 F2 CA| andn ecx,eax,edx
|C1 E2 68 F7 C8| bextr ecx,eax,edx
There is no general algorithm, you have to consult the manual of instructions individually.