Search code examples
c++windowsvisual-studiocrypto++

Crypto++ AES crashing in release mode using VS2017

I have the following code. A very simple example of encryption and decryption of string "abcd". I've tried it with sample from Crypto++ (https://www.cryptopp.com/wiki/CBC_mode) and it produces the same exception.

AutoSeededRandomPool rand;

SecByteBlock key(nullptr, AES::MAX_KEYLENGTH);
rand.GenerateBlock(key, key.size());

byte iv[AES::BLOCKSIZE];
rand.GenerateBlock(iv, AES::BLOCKSIZE);

std::string encryptedData;
CBC_Mode<AES>::Encryption cbcEncryption(key, key.size(), iv);

StringSource ss("abcd", true,
    new StreamTransformationFilter(cbcEncryption,
        new StringSink(encryptedData)
    )
);

std::string decryptedData;

CBC_Mode<AES>::Decryption cbcDecryption(key, key.size(), iv);

StringSource ss2(encryptedData, true,
    new StreamTransformationFilter(cbcDecryption,
        new StringSink(decryptedData)
    )
);

The problem is when I build in debug mode, it works fine but when I do it in release mode I get an exception from Crypto++ code ("StreamTransformationFilter: invalid PKCS #7 block padding found")

Exception

Solution

  • The problem is when I build in debug mode, it works fine but when I do it in release mode I get an exception from Crypto++ code ("StreamTransformationFilter: invalid PKCS #7 block padding found") ...

    It appears to be a compiler issue related to global optimizations. Our workaround was to disable global optimizations for the source file rijndael.cpp.

    In rijndael.cpp you can add the following around the top of the file to avoid the issue:

    #if defined(_MSC_VER) && (_MSC_VER >= 1910)
# pragma optimize("", off)
# pragma optimize("ts", on)
#endif

    You can reproduce the issue with the following in rijndael.cpp:

    #if defined(_MSC_VER) && (_MSC_VER >= 1910)
# pragma optimize("", off)
# pragma optimize("g", on)
#endif

    Also see Commit f57df06c5e6d and pragma optimize on MSDN.

    If your machine has AES-NI but you want to reproduce the issue, then comment-out the code that assigns g_hasAESNI in cpu.cpp. g_hasAESNI will retain the default value of false.

    --- a/cpu.cpp
+++ b/cpu.cpp
@@ -242,7 +242,7 @@ void DetectX86Features()
        g_hasSSSE3 = g_hasSSE2 && ((cpuid1[2] & (1<< 9)) != 0);
        g_hasSSE41 = g_hasSSE2 && ((cpuid1[2] & (1<<19)) != 0);
        g_hasSSE42 = g_hasSSE2 && ((cpuid1[2] & (1<<20)) != 0);
-       g_hasAESNI = g_hasSSE2 && ((cpuid1[2] & (1<<25)) != 0);
+       //g_hasAESNI = g_hasSSE2 && ((cpuid1[2] & (1<<25)) != 0);
        g_hasCLMUL = g_hasSSE2 && ((cpuid1[2] & (1<< 1)) != 0);