Search code examples
androidemailencryptionopensslbouncycastle

OpenSSL Encryption not Compatible with BouncyCastle Decryption?

I am sending a encrypted mail using OpenSSL in an android app. While sending the mail this is the relevant native code - 

cipher = EVP_rc2_40_cbc();
int flags = PKCS7_STREAM;
PKCS7* p7 = PKCS7_encrypt(certs, bio, cipher, flags);
int r = SMIME_write_PKCS7(out, p7, bio, flags);

The mail is successfully sent to the server and I get a sent mail notification.

Now when I am trying to open the mail using the another android app which has bouncycastle I am unable to open it. It says "Unable to decrypt message."

Also I tried openning the app using the webmail - interface on desktop. Here also I was unable to open the mail.

Any Idea what could be the problem?

I tried using a different cipher also ( 3des ). But that too did not work.

Update -> If I change my encryption code to bouncy castle then everything works fine. I am able to decrypt in both android app and web-browser. So is there a possibilty that this might be a OpenSSL Issue?

Update 2 -> I tried decrypting the encrypted text using openssl command prompt and it worked. So I guess it is surely some compatibilty problem? But is this a problem with bouncycastle or with openssl ?

Solution

  • It turns out I was missing a flag. 

    int flags = PKCS7_STREAM | PKCS7_BINARY;

    This did the charm.