Search code examples
c++cryptographyreleaseundefined-behaviorcrypto++

Cryptography - Bizzare behaviour in Release mode

I've got a project using Crypto++.

Crypto++ is a own project which builds in a static lib.
Aside from that I have another large project using some of the Crypto++ classes and processing various algorithms, which also builds in a static lib.

Two of the functions are these:

long long MyClass::EncryptMemory(std::vector<byte> &inOut, char *cPadding, int rounds)
{
    typedef std::numeric_limits<char> CharNumLimit;
    char sPadding = 0;
    //Calculates padding and returns value as provided type
    sPadding = CalcPad<decltype(sPadding)>(reinterpret_cast<MyClassBase*>(m_P)->BLOCKSIZE, static_cast<int>(inOut.size()));
    //Push random chars as padding, we never care about padding's content so it doesn't matter what the padding is
    for (auto i = 0; i < sPadding; ++i)
        inOut.push_back(sRandom(CharNumLimit::min(), CharNumLimit::max()));
    std::size_t nSize = inOut.size();
    EncryptAdvanced(inOut.data(), nSize, rounds);
    if (cPadding)
        *cPadding = sPadding;
    return nSize;
}

//Removing the padding is the responsibility of the caller.
//Nevertheless the string is encrypted with padding
//and should here be the right string with a little padding
long long MyClass::DecryptMemory(std::vector<byte> &inOut, int rounds)
{
    DecryptAdvanced(inOut.data(), inOut.size(), rounds);
    return inOut.size();
}

Where EncryptAdvanced and DecryptAdvanced pass the arguments to the Crypto++ object.

//...
AdvancedProcessBlocks(bytePtr, nullptr, bytePtr, length, 0);
//...

These functions have so far worked flawless, no modifications have been applied to them since months. The logic around them has evolved, though the calls and data passed to them did not change.

The data being encrypted / decrypted is rather small but has a dynamic size, which is being padded if (datasize % BLOCKSIZE) has a remainder.
Example: AES Blocksize is 16. Data is 31. Padding is 1. Data is now 32.

After encrypting and before decrypting, the string is the same - as in the picture.

Encrypt/Decrypt data

Running all this in debug mode apparently works as intended. Even when running this program on another computer (with VS installed for DLLs) it shows no difference. The data is correctly encrypted and decrypted.

Trying to run the same code in release mode results in a totally different encrypted string, plus it does not decrypt correctly - "trash data" is decrypted. The wrongly encrypted or decrypted data is consistent - always the same trash is decrypted. The key/password and the rounds/iterations are the same all the time.

Additional info: The data is saved in a file (ios_base::binary) and correctly processed in debug mode, from two different programs in the same solution using the same static librar(y/ies).

What could be the cause of this Debug / Release problem ?

I re-checked the git history a couple of times, debugged for days through the code, yet I cannot find any possible cause for this problem. If any information - aside from a (here rather impossible) MCVE is needed, please leave a comment.

Solution

  • Apparently this is a bug in CryptoPP. The minimum keylength of Rijndael / AES is set to 8 instead of 16. Using a invalid keylength of 8 bytes will cause a out-of-bounds access to the in-place array of Rcon values. This keylength of 8 byte is currently reported as valid and has to be fixed in CryptoPP.

    See this issue on github for more information. (On-going conversation)