I am currently working on an implementation of Bitslice DES for x64, and I would like to know how I could avoid 3-byte VEX prefixes as much as possible with the following AVX instructions:
vpor
vpxor
vpand
vpandn
I was told I should use 2-byte VEX prefixes as much as possible instead of 3-byte ones to make the code smaller and faster, but I don't know exactly how to do so. I read somewhere that I should use xmm[0-7] instead of xmm[89] and xmm1[0-5], but I don't know which combinations of operands would give me 2-byte VEX prefixes as opposed to 3-byte ones. These instructions are used with 2 or 3 operands as shown in the following snippet:
%macro sbox1 4
vmovdqa xmm7, xmm4
vpandn xmm4, xmm0
vpor xmm10, xmm5, xmm2
vpxor xmm13, xmm2, xmm0
vpxor xmm11, xmm5, xmm7
vpxor xmm14, xmm4, xmm3
vpandn xmm12, xmm13, xmm11
vpand xmm13, xmm10
vpxor xmm15, xmm11, xmm2
vpxor xmm8, xmm13, xmm3
vpandn xmm9, xmm14, xmm8
vpor xmm13, xmm5
vpor xmm5, xmm0
vpandn xmm8, xmm7
vpandn xmm15, xmm14
vpxor xmm13, xmm15
vpor xmm6, xmm5, xmm13
vpandn xmm5, xmm3
vpandn xmm15, xmm9, xmm13
vmovdqa xmm3, xmm8
vpxor xmm8, xmm6
vpxor xmm5, xmm3
vpand xmm13, xmm10
vpandn xmm4, xmm2
vpxor xmm2, xmm6, xmm14
vpxor xmm6, xmm10
vpandn xmm4, xmm2
vpxor xmm2, xmm4, pnot
vpxor xmm4, xmm11
vpxor xmm13, xmm2
vpor xmm4, xmm3
vpandn xmm2, xmm1, xmm8
vpor xmm14, xmm7
vpxor xmm4, xmm10
vpor xmm9, xmm1
vpxor xmm2, xmm13
vpxor xmm4, xmm0
vpxor xmm13, xmm4
vpxor xmm9, xmm13
vpor xmm5, xmm12
vpxor xmm9, %1
vmovdqa %1, xmm9
vpor xmm6, xmm5
vpor xmm13, xmm11
vpxor xmm6, xmm4
vpor xmm0, xmm1, xmm15
vpxor xmm13, xmm6
vpxor xmm2, %3
vmovdqa %3, xmm2
vpxor xmm13, %2
vpand xmm4, xmm15
vpandn xmm6, xmm14
vpxor xmm13, xmm0
vmovdqa %2, xmm13
vpxor xmm4, xmm6
vpor xmm4, xmm1
vpxor xmm4, xmm5
vpxor xmm4, %4
vmovdqa %4, xmm4
%endmacro
It turned out that I just needed to use xmm0...xmm7 as the last operand to have the instruction encoded with a 2-byte VEX prefix.