I'm trying to solve this question that wants me to encrypt a message by rotating the bits of each 10 bytes of a message to the left or to the right according to a certain key, for example:
key BYTE -2, 4, 1, 0, -3, 5, 2, -4, -4, 6
Where the sign indicates the direction of rotation, negative being to the left, positive to the right. The numbers indicate the magnitude of rotation. So the first byte of the message will be rotated twice to the left, the second 4 times to the right and so on and we do the same with the 11th and 12th byte and so on until the end of the message.
When I call this procedure, nothing happens to the message stored in memory, however:
EncryptR PROC USES ebx ecx esi,
ptrkey: PTR BYTE, ; pointer to the key array
ptrmessage: PTR BYTE, ; pointer to the plaintext message
mlength: DWORD ; length of the message array
; -----------------------------------------------------------------------------
mov ecx, mlength
inc ecx
mov edx, [ptrkey+10]
; mov edi, ptrmessage
L2:
mov esi, ptrkey
dec ecx
cmp ecx, 0
je endloop
L1:
push ecx
mov cl, [esi]
cmp cl,0
ja Skip1
neg cl
rol [ptrmessage], cl
jmp Skip2
Skip1:
ror [ptrmessage], cl
Skip2:
pop ecx
inc esi
inc ptrmessage
cmp [esi], edx
je L2
LOOP L1
endloop:
ret
I've tried using edi to point to the message but it doesn't work according to the syntax of the rotate instructions apparently. I've used ptrmessage directly for now so that I avoid build errors. I'm calling the procedure using invoke and have already declared it using the PROTO directive. I've tried doing it in a way which would reset the pointer of the key array to the first element once it is out of bounds while the message pointer keeps going.
MOV edx, offset Message1
call Strlength
Invoke EncryptR, ADDR key, addr Message1, eax
I'm not sure what's the problem, there's the possibilty of jumping to the endloop label prematurely which would explain it but it seems obvious that that wouldn't happen the first time round. Is there an issue with the way I have been using the rotate instructions?
There's a couple of issues. First of all ja is used for unsigned comparisons so it won't work for detecting values <0. Use jg (jump if greater) instead.
When you use Invoke MASM, does prepare a frame for you and passes parameters via stack, those are available inside the method, but if you see those ptrmessage in a debugger outside VS, those will be pointers to the addresses on the stack and not to the content of the message.
00401036 d3450c rol dword ptr [ebp+0Ch], cl
That's why you do not see and effect of the rol as it doesn't rol the bytes but the pointer. Not sure if MASM has the syntax to do this in a proper way, but you can always get the address to a register (say edi) and operate on that. In such case you would have to specify the size of data too (byte ptr).
Taking the following into consideration, I would write the EncryptR something like (changed lines makered with ;*)
EncryptR PROC USES ebx ecx esi,
ptrkey: PTR BYTE, ; pointer to the key array
ptrmessage: PTR BYTE, ; pointer to the plaintext message
mlength: DWORD ; length of the message array
; -----------------------------------------------------------------------------
mov ecx, mlength
inc ecx
mov edx, ptrkey
add edx, 10 ;*
mov edi, ptrmessage ;*
L2:
mov esi, ptrkey
dec ecx
cmp ecx, 0
je endloop
L1:
push ecx
mov cl, [esi]
cmp cl,0
ja Skip1 ;*
neg cl
rol byte ptr [edi], cl ;*
jmp Skip2
Skip1:
ror byte ptr [edi], cl ;*
Skip2:
pop ecx
inc esi
inc edi
cmp esi, edx
je L2
LOOP L1
endloop:
ret
EncryptR ENDP