I have the encode function for an ascii cipher, but need help to how to decode it

Question

I am working on a puzzle which involves decoding a cipher. I have been given the function used to encode the ciphertext, but I am not sure how I can reverse engineer this to find the decode function.

This is the function, in JavaScript:

function encodeText(ascii,x,y,z) {
    for (i=0; i<ascii.length; i++) {
        if(i%3==0){
            ascii[i]= (ascii[i]+x) % 256;
        }
        if(i%3==1){
            ascii[i]=(ascii[i]+y) % 256;
        }
        if(i%3==2){
            ascii[i]=(ascii[i]+z) % 256;
        }
    }
    return ascii;
}

I am aware that it returns an array of numbers, which is the cipher text I have been presented with.

Help is greatly appreciated!

Answer 1

This is a Vigenère Cipher with a key of length three (x, y, z).

ascii[i] and x are both smaller than 256, but (ascii[i]+x) might result in a number larger than 255. That is why the modulo operator flips it back into the range 0-255.

The decryption would look like (ascii[i] - x + 256) % 256.

ascii[i] - x is a direct opposite of ascii[i] + x, so I assume this is clear. ascii[i] may be a number smaller than x. In that case, ascii[i] - x might be less than zero. The problem is that -5 % 256 is -5 and not 251 (example). So, we need to move all possible results to the positive numbers that then apply the modulo operation.

For example, (74 - 80 + 256) is 250 and 250 % 256 is still 250. Furthermore, (84 - 80 + 256) is 260 and 260 % 256 is 4 which is the difference.

---

A Caesar Cipher and a Vigenère Cipher are usually broken by frequency analysis. Your instance of a Vigenère Cipher consists of three Caesar Ciphers. The key space is 256*256*256 = 16,777,216 which is too big to iterate over and read every possible recovered plaintext.

You should know what language the plaintext is in. If you have that, then you can split your ciphertext in three strings. The 1st, the 4th, the 7th and so on characters go into the first string. The 2nd, the 5th, the 8th and so on characters go into the second string. The 3rd, the 6th, the 9th and so on character go into the third string. Now, you can do on each of those three resulting strings a Frequency analysis and compare the results to the letter frequency of your target language (english example).

If the ciphertext is sufficiently long, you should be able to determine which encrypted letters are e and n. When you have that, the shift is linear. Let's say that the letter that you identified as e has an ordinal value of 43. If you look into an ASCII table, the actual lowercase e has an ordinal value of 101. That would mean that you have to calculate 43 - 101 = -58. Since this has to be positive, we can add 256 to get x = 198. You should repeat that process for y (second string) and z (third string), too.