How to return rsa encrypted string using openssl in jni

Question

In my android app, I am using openssl lib with JNI to encrypt the data. In the C program, I am able to encrypt the data successfully using the RSA_public_encrypt(). But I am not able to return the string using

return (*env)->NewStringUTF(env, encryptedData);

and I am getting the below error.

JNI DETECTED ERROR IN APPLICATION: input is not valid Modified UTF-8: illegal continuation byte 0x5c.

JNIEXPORT jstring JNICALL Java_com_jni_JniLayer_PublicEncryption
  (JNIEnv *env, jobject thisObj, jstring inData) {

__android_log_print(ANDROID_LOG_DEBUG, "Func", "--->  %s \n", __func__);
const char *cData = (*env)->GetStringUTFChars(env, inData, NULL);

if( NULL == cData )
    return (*env)->NewStringUTF(env, "NULL");

__android_log_print(ANDROID_LOG_DEBUG, "RSA", "The data to encrypt is : %s\n", cData);

RSA *rsa = NULL;
FILE *out = fopen("/sdcard/PublicKey", "r");
if( NULL == out )
{
    __android_log_print(ANDROID_LOG_ERROR, "File", "Unable to Open file : PublicKey\n");
    return (*env)->NewStringUTF(env, "NULL");
}
rsa = PEM_read_RSAPublicKey(out, NULL,NULL,NULL);
fclose(out);

//rsa = PEM_read_bio_RSA_PUBKEY(bioKey, &rsa, NULL, NULL);
if (NULL == rsa)
{
    __android_log_print(ANDROID_LOG_ERROR, "File", "Unable to generate RSA struct from BioKey\n");
    return (*env)->NewStringUTF(env, "NULL");
}

if(strlen(cData) > (RSA_size(rsa)-11) )
{
    __android_log_print(ANDROID_LOG_ERROR, "RSA", "The data to be encrypted is more than maximum length of data to encrypt is : %d.\n",(RSA_size(rsa)-11));
    return (*env)->NewStringUTF(env, "NULL");
}

unsigned char chEncryptedData[4098] = {};
int iResult = RSA_public_encrypt(strlen(cData), cData, chEncryptedData, rsa, RSA_PKCS1_PADDING);
(*env)->ReleaseStringUTFChars(env, inData, cData);

// If encryption fails, returns NULL string, else returns encrypted string
if(-1 == iResult)
{
    char *chErrMsg = malloc(256);
    ERR_load_crypto_strings();
    ERR_error_string(ERR_get_error(), chErrMsg);
    __android_log_print(ANDROID_LOG_ERROR, "RSA", "The data Encryption failed due to the reason : %s\n", chErrMsg);
    free(chErrMsg);
    return (*env)->NewStringUTF(env, "NULL");
}
__android_log_print(ANDROID_LOG_DEBUG, "RSA", "The Encrypted data is : %s\n", chEncryptedData);

return (*env)->NewStringUTF(env, chEncryptedData);
}

Answer 1

The problem is that you have not respected the difference between bytes and characters. In Java, bytes are simply eight-bit values and all are valid. A character on the other hand is a representation in UTF-8 format of a Unicode character. Now most byte values under 127 represent a UTF-8 character, but this is not the case for larger values. Characters can consist of several bytes and there are strict rules about which sequences form valid characters.

If you are encrypting a value, you need to work with bytes as there is no guarantee that your encryption will produce a valid character sequence. In fact it is almost certain that the sequence will not be valid.