How do I determine that the key or vector are incorrect using Rijndael to decrypt a file?

Question

When I use the wrong key, I am getting a "decrypted" garbage file and an exception from mscorlib.dll, "Specified block size is not valid for this algorithm." What I would like instead is for the decryption to fail entirely without throwing an exception.

Here's my current code (adapted from a vb.net example I found on the internet, so better solution would be appreciated if out there)

    public static bool EncryptOrDecryptFile(string strInputFile,
        string strOutputFile, string pKey, string pIv, CryptoAction Direction)
    {
        Byte[] bytKey = CreateKey(pKey);
        Byte[] bytIV = CreateIV(pIv);
        bool pRet = false;
        if (!File.Exists(strInputFile))
            return false;

        try
        {
            using (FileStream fsInput = new FileStream(strInputFile, FileMode.Open, FileAccess.Read))
            {
                using (FileStream fsOutput = new FileStream(strOutputFile, FileMode.OpenOrCreate, FileAccess.Write))
                {
                    fsOutput.SetLength(0);
                    byte[] bytBuffer = new byte[4097];
                    long lngBytesProcessed = 0;
                    long lngFileLength = fsInput.Length;
                    int intBytesInCurrentBlock = 0;
                    CryptoStream csCryptoStream = null;
                    RijndaelManaged cspRijndael = new RijndaelManaged();
                    cspRijndael.BlockSize = 4096;
                    switch (Direction)
                    {
                        case CryptoAction.ActionEncrypt:
                            csCryptoStream = new CryptoStream(fsOutput, cspRijndael.CreateEncryptor(bytKey, bytIV),
                                                              CryptoStreamMode.Write);
                            break;
                        case CryptoAction.ActionDecrypt:
                            csCryptoStream = new CryptoStream(fsOutput, cspRijndael.CreateDecryptor(bytKey, bytIV),
                                                              CryptoStreamMode.Write);
                            break;
                    }
                    while (lngBytesProcessed < lngFileLength)
                    {
                        intBytesInCurrentBlock = fsInput.Read(bytBuffer, 0, 4096);
                        csCryptoStream.Write(bytBuffer, 0, intBytesInCurrentBlock);
                        lngBytesProcessed = lngBytesProcessed + Convert.ToInt64(intBytesInCurrentBlock);
                    }
                    csCryptoStream.Close();
                }
            }
            pRet = true;
        }
        catch (Exception ex)
        {
            pRet = false;
        }
        return pRet;
    }
    #endregion
}

Answer 1

Given that any combination of bits of the right length could be a valid key, and the same is true for the IV, there's no way for the crypto stream to determine that either is incorrect until it tries to decrypt the final block of cypher text - at which point it detects that the padding is wrong and throws an exception. Also, you can't use any block size you want - there are certain block sizes that are valid for particular crypto providers - for Rijndael, I think valid block sizes are 128, 192 and 256.

The only way to avoid the "garbage" file would be to perform all of the decryption in memory, or write to a temporary file rather than the final output file. There's no way to avoid the exception, but it should be an unusual occurrence anyway.

---

I'd also get rid of all of the buffer management in your code, and length checks, etc, and just use Stream.CopyTo between your input stream and the crypto stream. E.g.:

using (FileStream fsInput = new FileStream(strInputFile, FileMode.Open, FileAccess.Read))
{
    using (FileStream fsOutput = new FileStream(strOutputFile, FileMode.OpenOrCreate, FileAccess.Write))
    {
        CryptoStream csCryptoStream = null;
        RijndaelManaged cspRijndael = new RijndaelManaged();
        cspRijndael.BlockSize = 256;
        switch (Direction)
        {
            case CryptoAction.ActionEncrypt:
                csCryptoStream = new CryptoStream(fsOutput, cspRijndael.CreateEncryptor(bytKey, bytIV),
                                                  CryptoStreamMode.Write);
                break;
            case CryptoAction.ActionDecrypt:
                csCryptoStream = new CryptoStream(fsOutput, cspRijndael.CreateDecryptor(bytKey, bytIV),
                                                  CryptoStreamMode.Write);
                break;
        }
        fsInput.CopyTo(csCryptoStream);
        csCryptoStream.Close();
    }
}
pRet = true;