Search code examples
c#encryptionrijndaelmanaged

RijndaelManaged Decryption Issue


I support a huge C++ system developed originally in the early 1990's. (I have mentioned this system in other posts of mine) We are in the process of developing all new C# applications to replace the older C++ systems. Using TcpClient, unencrypted or encrypted commands are sent to and from our C# applications to the C++ system. (encryption is a selectable feature). Unencrypted commands are both sent and received without issue. The encrypted commands are a different story. We can successfully encrypt our commands and the C++ system will accept those commands. However, encrypted messages sent FROM C++ TO our C# system, don't decrypt correctly. I have gotten a few decryptions to not fail, but they all result in a cryptic string. The encryption uses RijndaelManaged and a CryptoStream, that originated from the early 1990s. If anybody out there can get the decryption code to work, I'd say good things about you for many years to come :-) (if you are wondering why we don't replace the encryption style for something simpler, it is because several other older C++ systems still use this technique)

For sending the encrypted command, this works correctly:

        //Generic messages array
        var messages = new string[4];
        messages[0] = "";
        messages[1] = "Service Manager";
        messages[2] =
            "<COMMAND_BLOCK><OPERATOR User=\"1234\" Password=\"password\"/>" +
            "<COMMAND Target=\"targetServer\" Command=\"1024\" " +
            "Data=\"1\" Priority=\"HIGH\" Id=\"-1\" " +
            "Workstation=\"Server1\"/></COMMAND_BLOCK>{Convert.ToChar(10)}";
        messages[3] = IsEncryptionEnabled ? "1" : "0"; 


                var formattedMessage = $"{messages[1]}{Convert.ToChar(10)}{messages[2]}";
                formattedMessage =
                    $"{Convert.ToChar(messages[2].Length)}{Convert.ToChar((int)Math.Floor((double)messages[2].Length / 255))}{formattedMessage}";

                if (WebApiCommon.Globals.IsEncryptionEnabled) //Yes, encryption is crazy
                {
                    var commandBytes = Encoding.UTF8.GetBytes(messages[2]);
                    var messageLength = commandBytes.Length;
                    var allocatedLength = messageLength;
                    var blockLength = 16;

                    allocatedLength = ((messageLength + blockLength - 1) / blockLength) * blockLength;

                    var messageBytes = new byte[allocatedLength];

                    for (var x = 0; x < allocatedLength; x++)
                    {
                        if (x < messageLength)
                            messageBytes[x] = commandBytes[x];
                        else // pad out for encryption
                            messageBytes[x] = 0;
                    }

                    while (keyString.Length < 16) // make sure keystring is exactly 16 bytes.
                        keyString += keyString;
                    if (keyString.Length > 16)
                        keyString = keyString.Substring(0, 16);
                    var encoding = Encoding.GetEncoding(1252); // ANSI 1252

                    var key = encoding.GetBytes(keyString);

                    //create empty IV
                    var iv = new byte[16];
                    for (var x = 0; x < iv.Length; x++)
                        iv[x] = 0;

                    //create encryption object
                    var rijndael = new RijndaelManaged
                    {
                        BlockSize = 128,
                        KeySize = 128,
                        Padding = PaddingMode.None
                    };

                    //break into blockLength byte blocks and encrypt

                    var block = new byte[blockLength];
                    var encryptedBytes = new byte[allocatedLength];
                    for (var x = 0; x < allocatedLength; x += blockLength)
                    {
                        // copy current block
                        for (var j = 0; j < blockLength; j++)
                            block[j] = messageBytes[x + j];

                        using (var ms1 = new MemoryStream())
                        {
                            using (var cs1 = new CryptoStream(ms1, rijndael.CreateEncryptor(key, iv), CryptoStreamMode.Write))
                                cs1.Write(block, 0, block.Length);
                            var block1 = ms1.ToArray();
                            for (var j = 0; j < blockLength; j++)
                                encryptedBytes[x + j] = block1[j];
                        }
                    }
                    var lsbMessageLength = new byte[2];

                    lsbMessageLength[0] = (byte)(allocatedLength & 0xFF);
                    lsbMessageLength[1] = (byte)(((allocatedLength >> 8) & 0xF) + 0x40); // 0x40 = flag to enable encryption

                    var controlPortStream = controlConnection.GetStream();
                    controlPortStream.Write(lsbMessageLength, 0, lsbMessageLength.Length);
                    controlPortStream.Write(encryptedBytes, 0, encryptedBytes.Length);
                    controlPortStream.Flush();
                }

I am including all my decryption code attempts, commented out, along with the decryption code I would expect to work, but doesn't. If anybody can see what I'm doing wrong, please let me know (code preferred):

    private string DecryptMPString(string incomingMessage)
    {
        var keyString =
            WebApiCommon.ApiRequests.SysSettings.GetList(new SysSettingDto { SectionName = "Configuration", VariableName = "Site name" }).FirstOrDefault()?.Value;

        var encryptedBytes2 = Encoding.UTF8.GetBytes(incomingMessage);
        var encryptedBytes = ConvertHexToAscii(encryptedBytes2);

        var messageLength = encryptedBytes.Length;
        var allocatedLength = messageLength;
        var blockLength = 16;

        allocatedLength = ((messageLength + blockLength - 1) / blockLength) * blockLength;

        var messageBytes = new byte[allocatedLength];

        for (var x = 0; x < allocatedLength; x++)
        {
            if (x < messageLength)
                messageBytes[x] = encryptedBytes[x];
            else // pad out for encryption
                messageBytes[x] = 0;
        }

        while (keyString.Length < 16) // make sure keystring is exactly 16 bytes.
            keyString += keyString;
        if (keyString.Length > 16)
            keyString = keyString.Substring(0, 16);
        var encoding = Encoding.GetEncoding(1252); // ANSI 1252

        var key = encoding.GetBytes(keyString);

        //create empty IV
        var iv = new byte[16];
        for (var x = 0; x < iv.Length; x++)
            iv[x] = 0;

        //create encryption object
        //var rijndael = new RijndaelManaged
        //{
        //    BlockSize = 128,
        //    KeySize = 128,
        //    Padding = PaddingMode.None
        //};

        //break into blockLength byte blocks and encrypt

        //var block = new byte[blockLength];
        //var decryptedBytes = new byte[allocatedLength];
        //for (var x = 0; x < allocatedLength; x += blockLength)
        //{
        //    // copy current block
        //    for (var j = 0; j < blockLength; j++)
        //        block[j] = messageBytes[x + j];

        using (var rijndael = new RijndaelManaged())
        {
            rijndael.BlockSize = 128;
            rijndael.KeySize = 128;
            rijndael.Padding = PaddingMode.None;

            var decryptor = rijndael.CreateDecryptor(key, iv);

            var decryptedBytes = decryptor.TransformFinalBlock(messageBytes, 0, allocatedLength);

            return Encoding.UTF8.GetString(decryptedBytes, 0, decryptedBytes.Length);
        }

        //using (var encryptedStream = new MemoryStream(encryptedBytes))
        //    {
        //        using (var decryptedStream = new MemoryStream())
        //        {
        //            using (var cryptoStream = new CryptoStream(encryptedStream, rijndael.CreateDecryptor(key, iv), CryptoStreamMode.Read))
        //            {
        //                int data;
        //                while ((data = cryptoStream.ReadByte()) != -1)
        //                    decryptedStream.WriteByte((byte)data);
        //            }

        //            return Encoding.UTF8.GetString(decryptedStream.ToArray(), 0, decryptedStream.ToArray().Length);
        //            //var block1 = decryptedStream.ToArray();
        //        //for (var j = 0; j < blockLength; j++)
        //        //    decryptedBytes[x + j] = block1[j];
        //    }
        //}
        //}
        
        // return Encoding.UTF8.GetString(decryptedBytes, 0, decryptedBytes.Length);


        //using (var encryptedStream = new MemoryStream(encoding.GetBytes(incomingMessage)))
        //{
        //    //stream where decrypted contents will be stored
        //    using (var decryptedStream = new MemoryStream())
        //    {
        //        using (var aes = new RijndaelManaged { KeySize = 128, BlockSize = 128, Padding = PaddingMode.None })
        //        {
        //            using (var decryptor = aes.CreateDecryptor(key, iv))
        //            {
        //                //decrypt stream and write it to parent stream
        //                using (var cryptoStream = new CryptoStream(encryptedStream, decryptor, CryptoStreamMode.Read))
        //                {
        //                    int data;

        //                    while ((data = cryptoStream.ReadByte()) != -1)
        //                        decryptedStream.WriteByte((byte)data);
        //                }
        //            }
        //        }

        //        //reset position in prep for reading
        //        decryptedStream.Position = 0;
        //        var decryptedBytes = decryptedStream.ToArray();
        //        return Encoding.UTF8.GetString(decryptedBytes, 0, decryptedBytes.Length);
        //    }
        //}

    }

    private static byte[] ConvertHexToAscii(byte[] hexBytes)
    {
        int newLength = (hexBytes.Length) / 2;
        var buffer = new byte[newLength];
        for (int i = 0, j = 0; i < newLength; i++, j += 2)
            buffer[i] = (byte)(((hexBytes[j] - '0') << 4) + (hexBytes[j + 1] - '0'));

        return buffer;
    }

This is days after the initial question, but here is the C++ Encrypt code used for creating the message that is picked up by my C# solution:

  BYTE* CEncrypt::Encrypt(LPCTSTR pszData,WORD& nLength)
  {
    if(nLength && m_bEncryptEnabled && m_nBlockSize == 16)
    {
        int nBufferSize = m_nBlockSize * (nLength/m_nBlockSize + (nLength %       m_nBlockSize == 0?0:1));
        BYTE* cBuffer = new BYTE[nBufferSize];
        if(cBuffer)
        {
            memcpy(cBuffer,pszData,nLength);
            while (nLength < nBufferSize)   // zero last bytes in case short
                    cBuffer[nLength++] = '\0';
            for (int nIndex = 0; nIndex < nLength; nIndex += m_nBlockSize)
            {
                Encrypt(cBuffer + nIndex);      
            }
        return cBuffer;
        }
    }
    return NULL;
  }

  /* There is an obvious time/space trade-off possible here.     *
   * Instead of just one ftable[], I could have 4, the other     *
   * 3 pre-rotated to save the ROTL8, ROTL16 and ROTL24 overhead */ 
  void CEncrypt::Encrypt(BYTE *buff)
  {
      int i,j,k,m;
      DWORD a[8],b[8],*x,*y,*t;

      for (i=j=0; i<Nb; i++,j+=4)
      {
          a[i] = pack((BYTE *)&buff[j]);
          a[i] ^= fkey[i];
      }
      k = Nb;
      x = a;
      y = b;

    /* State alternates between a and b */
      for (i=1; i<Nr; i++)
      { /* Nr is number of rounds. May be odd. */

        /* if Nb is fixed - unroll this next loop and hard-code in the values       of fi[]  */

          for (m=j=0; j<Nb; j++,m+=3)
          { /* deal with each 32-bit element of the State */
            /* This is the time-critical bit */
              y[j]=fkey[k++]^ftable[(BYTE)x[j]]^
                   ROTL8(ftable[(BYTE)(x[fi[m]]>>8)])^
                   ROTL16(ftable[(BYTE)(x[fi[m+1]]>>16)])^
                   ROTL24(ftable[x[fi[m+2]]>>24]);
          }
          t = x;
          x = y;
          y = t;      /* swap pointers */
      }

    /* Last Round - unroll if possible */ 
      for (m=j=0; j<Nb; j++,m+=3)
      {
          y[j] = fkey[k++]^(DWORD)fbsub[(BYTE)x[j]]^
               ROTL8((DWORD)fbsub[(BYTE)(x[fi[m]]>>8)])^
               ROTL16((DWORD)fbsub[(BYTE)(x[fi[m+1]]>>16)])^
               ROTL24((DWORD)fbsub[x[fi[m+2]]>>24]);
      }   

      for (i=j=0; i<Nb; i++,j+=4)
      {
          unpack(y[i], (BYTE *)&buff[j]);
          x[i] = y[i] = 0;   /* clean up stack */
      }
      return;
  }

Solution

  • The problem is that your encrypt code creates a new encryptor for each block:

    for (var x = 0; x < allocatedLength; x += blockLength)
    {
        // copy current block
        for (var j = 0; j < blockLength; j++)
            block[j] = messageBytes[x + j];
    
        using (var ms1 = new MemoryStream())
        {
            using (var cs1 = new CryptoStream(ms1, rijndael.CreateEncryptor(key, iv), CryptoStreamMode.Write))
                cs1.Write(block, 0, block.Length);
            var block1 = ms1.ToArray();
            for (var j = 0; j < blockLength; j++)
                encryptedBytes[x + j] = block1[j];
        }
    }
    

    ... which makes it essentially an ECB transform. The C++ code is doing the same thing.

    On your C# Decrypt, you attempt to decrypted the whole block at once:

    using (var rijndael = new RijndaelManaged())
    {
        rijndael.BlockSize = 128;
        rijndael.KeySize = 128;
        rijndael.Padding = PaddingMode.None;
        var decryptor = rijndael.CreateDecryptor(key, iv);
        var decryptedBytes = decryptor.TransformFinalBlock(messageBytes, 0, allocatedLength);
        return Encoding.UTF8.GetString(decryptedBytes, 0, decryptedBytes.Length);
    }
    

    ...which will work for the first block, but not the rest. Change it to process each block with a new decryptor...and it will work:

    string finalString = "";
    var blockLength = 16;
    using (var rijndael = new RijndaelManaged())
    {
        rijndael.BlockSize = 128;
        rijndael.KeySize = 128;
        rijndael.Padding = PaddingMode.None;
        for (var bytePos = 0; bytePos < allocatedLength; bytePos += blockLength)
        {
            var decryptor = rijndael.CreateDecryptor(key, iv);
            var decryptedBytes = decryptor.TransformFinalBlock(messageBytes.Skip(bytePos).ToArray(), 0, blockLength);
            finalString += Encoding.UTF8.GetString(decryptedBytes, 0, decryptedBytes.Length);
        }
    }
    return finalString;