I know CryptoStream's FlushFinalBlock method throws when the padding is wrong which likely means the key was wrong.
What I'm trying to figure out is whether this is a bullet-proof method of testing the authenticity of the key itself, i.e, whether FlushFinalBlock is guaranteed to throw if given the wrong key. Or, put differently, is it possible for the wrong key to yield a result that throws no exception but is just garbled output? If it is possible what is the rough likelihood of this? (I'm asking about AES256 specifically).
Assume I don't know what the encrypted data is and have no control over its encryption, so I have no way to validate it other than through a decryption attempt.
To answer the question straight away: Decrypting a ciphertext with an wrong key, i.e. a key that was not used in the encryption, does not necessarily result in an exception.
The following C# code shows the decryption of a ciphertext with three keys (the ciphertext was generated with AES/CBC and PKCS#7 padding):
using System;
using System.IO;
using System.Security.Cryptography;
using System.Text;
...
static void Main(string[] args)
{
byte[] iv = Convert.FromHexString("4B4D907F815EAB0EA3E8A2140968C395");
byte[] ciphertext = Convert.FromHexString("4A9BE2E236868EC0D04F0D280A2876920F79C10969F32D751FF1976E6446F7BBF2469957130E4EE1CC56C386426E1C5C");
// 1.
testCase(1, Convert.FromHexString("5DBF2259D534802EA0C8B24FD6CA876C345AE4C4AE2E000BCB05B33E7465FAEF"), iv, ciphertext);
// 2.
testCase(2, Convert.FromHexString("DFAC3EDECC5F53EF1ACFEE07085A2A5C4327D1057BED2405D5E18EF12DE05C63"), iv, ciphertext);
// 3.
testCase(3, Convert.FromHexString("000102030405060708090A0B0C0D0E0F000102030405060708090A0B0C0D0E0F"), iv, ciphertext);
}
private static void testCase(int num, byte[] key, byte[] iv, byte[] ciphertext)
{
Console.WriteLine("" + num + ".");
try
{
byte[] decryptedNone = Decrypt(ciphertext, key, iv, PaddingMode.None);
byte[] decryptedPkcs7 = Decrypt(ciphertext, key, iv, PaddingMode.PKCS7);
Console.WriteLine("before depadding: " + Convert.ToHexString(decryptedNone));
Console.WriteLine("after depadding: " + Convert.ToHexString(decryptedPkcs7));
Console.WriteLine("UTF-8 decoded: " + Encoding.UTF8.GetString(decryptedPkcs7));
Console.WriteLine();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
private static byte[] Decrypt(byte[] cipherText, byte[] Key, byte[] IV, PaddingMode pm)
{
byte[] plaintext = null;
using (Aes aesAlg = Aes.Create())
{
aesAlg.Key = Key;
aesAlg.IV = IV;
aesAlg.Padding = pm;
ICryptoTransform decryptor = aesAlg.CreateDecryptor(aesAlg.Key, aesAlg.IV);
using (MemoryStream msDecrypt = new MemoryStream(cipherText))
{
using (CryptoStream csDecrypt = new CryptoStream(msDecrypt, decryptor, CryptoStreamMode.Read))
{
using (MemoryStream plainTextStream = new MemoryStream())
{
csDecrypt.CopyTo(plainTextStream);
plaintext = plainTextStream.ToArray();
}
}
msDecrypt.Close();
}
}
return plaintext;
}
The 1st decryption uses the correct key. The last 5 bytes of the decrypted plaintext are 0x05050505. This is compliant with PKCS#7, i.e. the padding is valid and will be removed automatically. The decryption is successful, the plaintext is the same as the original plaintext.
The 2nd decryption uses a wrong key. By chance the last 2 bytes of the decrypted plaintext are 0x0202, which is also compliant with PKCS#7. Therefore this data is interpreted as valid padding and removed automatically. No exception is thrown. Of course, the decryption is not successful because the plaintext does not match the original plaintext.
The 3rd decryption also uses a wrong key. Here the last bytes are not compliant with PKCS#7. Therefore this is interpreted as invalid padding and a corresponding exception is thrown (CryptographicException: Padding is invalid and cannot be removed).
Accordingly, the output is:
1.
before depadding: 54686520717569636B2062726F776E20666F78206A756D7073206F76657220746865206C617A7920646F670505050505
after depadding: 54686520717569636B2062726F776E20666F78206A756D7073206F76657220746865206C617A7920646F67
UTF-8 decoded: The quick brown fox jumps over the lazy dog
2.
before depadding: 4D5B5017ADF3B5EFB5716AB2CCFF8BA6129E35A123287CD564F12C2401AB985EEBC7E274B685595251059C56E2BC0202
after depadding: 4D5B5017ADF3B5EFB5716AB2CCFF8BA6129E35A123287CD564F12C2401AB985EEBC7E274B685595251059C56E2BC
UTF-8 decoded: M[P↨???qj?????↕?5?#(|?d?,$☺??^???t??YRQ♣?V?
3.
Padding is invalid and cannot be removed.
What are the probabilities for 2., i.e. for the case that a wrong key does not cause an exception? For PKCS#7 the value of the padding bytes corresponds to the number of padding bytes, i.e. the following paddings are possible for AES with a block size of 16 bytes, see PKCS#7 for details:
0x01, 0x0202, 0x030303, ..., 0x10101010101010101010101010101010 (full block)
The probability, for a 0x01 at the end is 265-1, for 0x0202 265-2, etc. up to 265-16 for the full block. I.e. especially the contribution of the small padding bytes to the probability is not negligible.
The above probabilities apply under the assumption that all padding bytes are checked (which may not be true for all implementations). If fewer padding bytes are checked, the probability increases, see the other answer.
So far, block cipher modes (such as CBC) have been considered, which generally require mandatory padding. There are also stream cipher modes (such as CTR) that do not require padding. For these, no exception is generally thrown because of the missing padding.
As noted in the comment and elaborated in the other answer, authenticated encryption/decryption (e.g. GCM) provides a reliable way to detect whether the ciphertext has been tampered with and also whether the correct key is being used.