usage of RSA_NO_PADDING when decrypting data with RSA private key using openssl libcrypto

Question

I generated RSA private key and public key as below,

openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048
openssl rsa -pubout -in pri.key -out pub.key

And encrypted text file as below,

openssl pkeyutl -encrypt -pubin -inkey ~/pub.key -in ~/1.txt -out ~/1e.txt

Then I wrote below program to decrypt the encryted file. However, it seemed that decrypt didn't work as expected.

#include <openssl/evp.h>
#include <openssl/rsa.h>
#include <openssl/pem.h>
#include <openssl/err.h>
#include <openssl/conf.h>
#include <iostream>

using namespace std;

void
cleanup()
{
    EVP_cleanup();
    CRYPTO_cleanup_all_ex_data();
    ERR_free_strings();
}

int
main(int argc, char** argv)
{
    ERR_load_crypto_strings();
    OpenSSL_add_all_algorithms();
    OPENSSL_config(nullptr);

    cout<<"Initialize crypto library done"<<endl;

    EVP_PKEY * key = EVP_PKEY_new();
    if (key == nullptr) {
        cout<<"Failed to contruct new key"<<endl;
        return 1;
    }
    FILE * fpri = nullptr;
    fpri = fopen("/home/stack/pri.key", "r");
    if (fpri == nullptr) {
        cout<<"Failed to load private key"<<endl;
        return 1;
    }
    key = PEM_read_PrivateKey(fpri, &key, nullptr, nullptr);
    if (key == nullptr) {
        std::cout<<"Read private key failed"<<endl;
        return 1;
    }
    cout<<"load private key successfully"<<endl;
    EVP_PKEY_CTX *ctx = nullptr;
    ctx = EVP_PKEY_CTX_new(key, nullptr);
    EVP_PKEY_decrypt_init(ctx);
    EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_NO_PADDING);

    size_t outlen = 0, inlen = 0;
    unsigned char * out = nullptr, * in = nullptr;

    char buf[1024];
    FILE * fe = nullptr;
    fe = fopen("/home/stack/1e.txt", "r");
    size_t len = fread(buf, 1, sizeof(buf),  fe);
    cout<<"data input length is "<<len<<endl;
    EVP_PKEY_decrypt(ctx, NULL, &outlen, in, inlen);
    cout<<"outlen is "<<outlen<<endl;

    out = (unsigned char*)OPENSSL_malloc(outlen);
    EVP_PKEY_decrypt(ctx, out, &outlen, in, inlen);
    cout<<"decrypted data "<<out<<endl;
    cleanup();

    return 0;

}

When executing the code, the result is as below,

[stack@agent ~]$ ./test
Initialize crypto library done
load private key successfully
data input length is 256
outlen is 256
decrypted data

It seemed the decrypted data length was not correct and not printable.

When I commented out the instruction "EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_NO_PADDING);", it worked well.

I also tried with RSA_PKCS1_OAEP_PADDING, it didn't work either. If RSA PADDING is not set, it worked.

My question is as below,

1. Which padding is used in following command?

   openssl pkeyutl -encrypt -pubin -inkey ~/pub.key -in ~/1.txt -out ~/1e.txt
   

2. Is padding necessary for RSA encrypt/decrypt? If so, how could I apply the padding mechanism?

Answer 1

You should use EVP_PKEY_CTX_set_rsa_padding if you use not default padding in openssl pkeyutl encryption. See openssl pkeyutl documentation for details about -rsa_padding_mode.