Different aes-128-cbc between C# and openssl

Question

I'm trying to decrypt this data, given a key and an IV, but the result is different between c# and openssl

Openssl operation:

### client key
$ hexkey=f656d037b173ef3e11169f27231a84b6
### IV for this record
$ hexiv=404142434445464748494a4b4c4d4e4f
### encrypted data
$ echo '22 7b c9 ba 81 ef 30 f2 a8 a7 8f f1 df 50 84 4d'  > /tmp/msg1
$ echo '58 04 b7 ee b2 e2 14 c3 2b 68 92 ac a3 db 7b 78' >> /tmp/msg1
$ echo '07 7f dd 90 06 7c 51 6b ac b3 ba 90 de df 72 0f' >> /tmp/msg1
$ xxd -r -p /tmp/msg1 \
  | openssl enc -d -nopad -aes-128-cbc -K $hexkey -iv $hexiv | hexdump

C# operation:

 public static string DecryptWithOpenSSL(byte[] base64CipherText, byte[] key, byte[] iv)
        {
            try
            {
                using (Aes aesAlg = Aes.Create())
                {
                    aesAlg.Key = key;
                    aesAlg.IV = iv;
                    aesAlg.Mode = CipherMode.CBC;
                    aesAlg.Padding = PaddingMode.None;

                    ICryptoTransform decryptor = aesAlg.CreateDecryptor(aesAlg.Key, aesAlg.IV);

                    using (MemoryStream msDecrypt = new MemoryStream(base64CipherText))
                    using (CryptoStream csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read))
                    using (StreamReader srDecrypt = new StreamReader(csDecrypt))
                    {
                        return srDecrypt.ReadToEnd();
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error decrypting data: " + ex.Message);
                return null;
            }
        }

Openssl result: 14 00 00 0c cf 91 96 26 f1 36 0c 53 6a aa d7 3a ....... C# result: 14 00 00 0C CF 91 EFBFBD26EFBFBD360C536AEFBFBDEFBFBD3AEFBFBD

The key and IV are in the correct format. As you can see, only first 6 bytes looks the same. If anyone could help :)

Answer 1

The other answer will likely help you get past the bump. But if you're interested, I'll try to explain what your strange C# result means.

If you squint hard at your 2 different outputs, you'll see that they actually line up nicely like so:

(SSL) 14 00 00 0C CF 91 [96    ] 26 [F1    ] 36 0C 53 ...
(C#)  14 00 00 0C CF 91 [EFBFBD] 26 [EFBFBD] 36 0C 53 ...

Here's a useful tip: If you ever see 0xEFBFBD, you should stop and immediately suspect that you've run into a UTF-8 encoding issue. It is what's known as the Unicode REPLACEMENT CHARACTER (scroll down till you get to UTF-8 and you'll see the magic sequence).

StreamReader is designed to read strings, and it defaults to UTF-8 encoding unless specified otherwise. And that is why you're seeing what you are seeing.

Without getting bogged down on the encoding process (Wikipedia's explanation isn't the best but it'll do), both your problematic characters (96 and F1) are not meant to sit where they are in a valid UTF-8 sequence.

The sequence you have 0xCF/91/96 is 11001111/10010001/10010110 in binary. CF starts with 110 which means it's the first of a 2-byte sequence. The next byte, 91 starts with 10 meaning it's a continuation (second part of the 2-byte sequence).
The next byte (your problematic character 96) also starts with 10 - supposedly another "continuation byte". But we're not expecting another continuation! We have completed the 2-byte sequence. So the decoder freaks out and throws in your magic EFBFBD/�/REPLACEMENT CHARACTER in its place.

Similarly, F1 is 11110001 which is meant to be the first of a 4-byte sequence. But none of the following bytes have the "continuation" marker. Another freak out.

So yeah, character encoding is a funny -but interesting- rabbit hole to fall into.