How to revert an Encryption process

Question

As a task, I've been given some assembly code that, takes an input from the user (encryption key) and then a string of alphanumeric characters which it then encrypts by simply scrambling the the order of the bytes in the EAX, EDX and EXD registers (as far as I understand it.)

My task is I've go to take the encrypted string and decrypt it back to the original string input.

I first attempted to invert the steps of the encryption process (i.e. changing rol to ror, etc.) but soon realised that this didn't work.

If somebody could spend the time explaining how a decryption process / code could be made, then I'd be extremely grateful.

    __asm {
    encryptX:  push ecx
        xchg eax, ecx               
        neg  al                     
        ror  al, 1                  
        xor  al, byte ptr[ecx]      
        push edx                    
        mov  edx, eax               
        xchg eax, ecx               
        rol  byte ptr[eax], 3       
        xor  dl, byte ptr[eax]      
        rol  dl, 2                  
        mov  eax, edx               
        pop  edx                    
        pop  ecx                    
        ret                         
    }

Answer 1

You didn't post it, but since this is the third time someone is requesting this exact kind of exercise to be solved, I assume that EAX holds a pointer to the current key char and ECX hold the current message char (i.e. the char to be encrypted).

1. Reverse engine the algorithm

This is a very simple stateless algorithm, I annotated every useful instructions.
I assume you understand assembly, if don't you can take a tutorial online.
There is nothing more to add to this reverse enginering, the code is equivalent (or even more clear due to the vertical splitting into simple operations) to the high level source code in simplicity

;EAX = ptr to current key char, k
;ECX = current message char, c

encryptX:  
        push ecx

        xchg eax, ecx             ;EAX = c, ECX = ptr k
 
        neg  al                   ;AL = NEG(c)       
        ror  al, 1                ;AL = ROL(NEG(c), 1)
        xor  al, byte ptr[ecx]    ;AL =  ROL(NEG(c), 1) xor k
   
        push edx                    

        mov  edx, eax             ;EDX =  ROL(NEG(c), 1) xor k    
        xchg eax, ecx             ;EAX = ptr k, ECX = ROL(NEG(c), 1) xor k 

        rol  byte ptr[eax], 3     ;K = ROL(k, 3)  
        xor  dl, byte ptr[eax]    ;DL = (ROL(NEG(c), 1) XOR k)  XOR ROL(k, 3)  
        rol  dl, 2                ;DL = ROL((ROL(NEG(c), 1) XOR k)  XOR ROL(k, 3), 2)

        mov  eax, edx         ;Return
             
        pop  edx                    
        pop  ecx                    
        ret  

Algorithm is e = ROL((ROR(NEG(c), 1) XOR k) XOR ROL(k, 3), 2)

I denote with e the scrambled char, with c the original char and with k the key char.

2. Invert the function

You must know the inverse of every function used, this is trivial, but here is the list

 f     |  f^-1
-------+-------
  XOR  |  XOR
  NEG  |  NEG
  ROL  |  ROR
  ROR  |  ROL

Then you start from the outern function toward the inner, remembering that you aim at c.

0. e = ROL((ROR(NEG(c), 1) XOR k)  XOR ROL(k, 3), 2)

1. ROR(e, 2) = (ROR(NEG(c), 1) XOR k)  XOR ROL(k, 3)

2. ROR(e, 2) XOR ROL(k, 3) = ROR(NEG(c), 1) XOR k

3. ROR(e, 2) XOR ROL(k, 3) XOR K = ROR(NEG(c), 1)

4. ROL(ROR(e, 2) XOR ROL(k, 3) XOR K, 1) = NEG(c)

5. NEG(ROL(ROR(e, 2) XOR ROL(k, 3) XOR K, 1)) = c

Descrambling is c = NEG(ROL(ROR(e, 2) XOR ROL(k, 3) XOR K, 1)).

3. Implement in assembly

You can easily code the expression above in assembly.

---

Warning

Double check the statements above, I didn't run the code nor made any test.
You should check for minor corrections, use your critical sense, don't copy paste any formula.
If you blindly believe other one code, then it's up to you if you fail your test.

---

May I ask you what is the context of this exercise? Is it an assignment from your assembly programming course?