I'm trying to implement an algorithm in x86 assembly which can encrypt a given hex value and then decrypt it in less then 20 to 30 lines, But to no avail.
I haven't really found any sort of algorithm which does such thing and in assembly as well, hopefully anyone of you guys have an idea or an algorithm that I can use.
I thought about decrypting each letter individually, but the only real encryption that I know that I might be able to implement is Caesar Cipher. But it's way to easy to decrypt for the thing I'm going to do.
I'm using the emu8086 emulator for this.
So you want a symmetric cipher, where the same key is needed to encrypt and decrypt? (i.e. the kind of crypto you'd use after establishing a shared key using Diffie-Hellman key exchange, or public-key crypto like RSA, or by some pre-existing secret side channel).
Obviously if you're designing it yourself you don't expect it to be actually secure, just hopefully non-trivial for an amateur to break by hand if you get it right and the application/protocol that uses it doesn't screw it up (e.g. leak the key, reuse keys insecurely, or generate weak keys). Real digital cryptography is hard; there's a reason the world spent years evaluating ciphers before standardizing on Rijndael as the winner to become AES in 2001, and why it's so complicated and works in chunks of 128 bits. And using it to encrypt/decrypt longer messages requires choosing one of various techniques to apply that block cipher without just repeating exactly the same operation on each 128-bit chunk separately, e.g. cipher-block chaining. (Then an attacker could tell if the plaintext had two 128-bit chunks that were identical or not.)
The simplest approach is to XOR the plaintext with some key. Doing that again will decrypt because that's how XOR works.
If you repeat the same short key for every byte or word of the message, that's the digital equivalent of a Caesar cipher; pretty easy for even amateurs to break.
If you have a secret key that's as long as the message, then this is a "one time pad". (Unless you use the same key with different plaintexts, then you've defeated the security because an attacker can XOR the two ciphertexts together to cancel the key, and have the XOR of two plaintexts. A chosen-plaintext attack would totally defeat this.) But if you do never reuse a truly random key, then this is the only cryptography that's provably secure. Of course it's not very useful because you need the sender and receiver to already have enough shared-secret data. Useful if you're sending a spy out into the field with one copy of a codebook, not useful for separate computers running indefinitely.
So simple XOR with a giant key is perfect if you never reuse keys, but much weaker than more complex encryption if you want to reuse a key or have a short key.
A good strategy might be to use a PRNG to expand a small key into an arbitrary-length stream of bytes which you XOR the message with. i.e. the key is a seed for a random number generator. (And is a separate input to the program, or one specific key could be hard-coded into a toy program.)
That's how Stream Ciphers like RC4 / RC5 work, i.e. they're CSPRNGs: Cryptographically-Secure Pseudo-Random Number Generators.
Just like one-time pads, they are totally defeated by reusing the same key, if the attacker knows this is happening. But obviously if you're designing your own cipher you shouldn't have any expectation of real security. Still, hard-coding a key into your program would be worse for this strategy compared to other ways of doing some mixing.
For this toy program, you can use a simple PRNG that isn't cryptographically secure, e.g. XORshift+. A compact xorshift32+ should be possible with a couple registers on 16-bit 8086. The main site I linked only has 128 and 256-bit state versions, but does mention xorshift32+ on a 16-bit CPU.
However, any old simple PRNG might be ok for your use, maybe even a simple LCG as a multiply / add with a constant, with modulo being an implicit modulo by 2^16 by only taking the 16-bit result in AX. That's only a couple instructions. If you cared about performance on a real 8086 you'd avoid multiply (because it's slow).
The other major strategy to consider is a sequence of rotate, XOR, and ADD, like the now-obsolete DES. https://en.wikipedia.org/wiki/Data_Encryption_Standard. (And maybe shift, but rotate and XOR are easily reversible because they don't destroy information.)
A simplified and maybe smaller version of that (e.g. using 32-bit chunks so you can rotate by one with only a pair of adc ax,ax / adc dx,dx instructions) might work ok.
There is a Tiny Encryption Algorithm (wikipedia has a C version) which is based on Feistel rounds (shift two ways, add magic constants, xor together), and is intended to be cheap + simple to implement, although some weaknesses have been found in it. (Although it does have 128-bit keys and a 64-bit block-size.) Broken attempt at a 32-bit x86 asm implementation in this question.
Shifting by 4 or 5 on 8086 is not cheap (need a count in CL, unless you can abuse AAA or something to split 4-bit nibbles with something cheaper than aam 16 which will do hardware division and is even slower). Doing far fewer than the suggested 64 rounds per block would help performance but not necessarily code-size, if you don't care about the full security.
Encrypting different messages with the same key should be less trivial for amateurs to break with a cipher of this kind of design than a stream cipher. And possibly even non-trivial for experience cryptographers.
You could still evolve the key every step using a PRNG like xorshift+ or something even simpler, so the same 4 letters don't encrypt the same way every time if they happen to be aligned. Or if you don't care about that and need the code-size to stay small, maybe you give up on that. Or just increment the key every step, or do cipher block chaining where you XOR the plaintext for a new block with the previous ciphertext. That's just one section of a Wiki article about various ways to use block cipher algos on a large stream of data.
Decrypt and encrypt are not the same function with this strategy; you need to do things in reverse order.