Search code examples
c++encryptioncrypto++

Crash when trying to encrypt a file with Crypto++ RSA Scheme

I have successfully used this some lines ago in my program:

string tmp;
    StringSource(msg, true, new PK_EncryptorFilter(*rng, *encryptor, new CryptoPP::HexEncoder(new StringSink(tmp))));
    return tmp;

So you know that the Crypto++ objects are well created and so.

Now I want to encrypt a whole binary file and save it to an adjacent file:

FileSource(file.c_str(), true, new PK_EncryptorFilter(*rng, *encryptor, new FileSink((file+".xx").c_str(), true)),true);

But this last line crashes with a debug error stating that abort() has been called.

Hunting down the error, I tried to change the second argument to the FileSource call to false, leading to the following code:

FileSource(file.c_str(), false, new PK_EncryptorFilter(*rng, *encryptor, new FileSink((file+".xx").c_str(), true)),true);

And then the error gone, but the destination file weights 0 bytes, nothing was read/wrote.

I do not know what can can the key to the problem, so, I hope someone can help a little bit.

EDIT: I am using Visual Studio 2013 Pro.

EDIT2: I hunted the error further.

This works and the file binary content is correctly printed on screen:

string s;
FileSource file2("C:\\test.jpg", true, new StringSink(s));
std::cout << s << std::endl;

But this don't work and ends with the mentioned crash.

string s;
FileSource file2("C:\\test.jpg", true, new PK_EncryptorFilter(*rng, *encryptor, new StringSink (s)));
std::cout << s << std::endl;

This is so strange since the same PK_EncryptorFilter filter is used in another method without trouble, as I stated at the beginning of the post.

Anyway, I post here my entire class, so as to get a clear idea of what is going on:

RSASystem::RSASystem()
{
    std::string pubkey = "...OMITED...";

    rng = new AutoSeededRandomPool;

    CryptoPP::HexDecoder decoder;
    decoder.Put((byte*)pubkey.c_str(), pubkey.size());
    decoder.MessageEnd();

    CryptoPP::HexDecoder decoder2;
    decoder2.Put((byte*)pubkey.c_str(), pubkey.size());
    decoder2.MessageEnd();

    verifier = new RSASSA_PKCS1v15_SHA_Verifier;
    encryptor = new RSAES_OAEP_SHA_Encryptor;

    verifier->AccessKey().Load(decoder);
    encryptor->AccessKey().Load(decoder2);
}

string RSASystem::encrypt(string msg)
{
    string tmp;
    StringSource(msg, true, new PK_EncryptorFilter(*rng, *encryptor, new CryptoPP::HexEncoder(new StringSink(tmp))));
    return tmp;
}

void RSASystem::encryptFile(string file)
{
    FileSource(file.c_str(), true, new PK_EncryptorFilter(*rng, *encryptor, new FileSink((file+".xx").c_str(), true)),true);
}

EDIT 3: After surrounding the code with try..catch() I got this error:

RSA/OAEP-MGF1(SHA-1): message length of 490986 exceeds the maximum of 214 for this public key

Which now I think can be easily solved.

Solution

  • I had already pending the encryption and security subject so I wasn't aware of the limitation on the length of the message of the RSA scheme. https://security.stackexchange.com/questions/44702/whats-the-limit-on-the-size-of-the-data-that-public-key-cryptos-can-handle

    So the solution passes by implementing an Integrated or Hybrid Encryption Scheme, like ECIES.

    I've done this successfully with Crypto++ using: http://www.cryptopp.com/wiki/Elliptic_Curve_Integrated_Encryption_Scheme

    Thanks to jww to point to the correct decision.