C++ OpenSSL RSA_free give Segmentation fault if EVP_PKEY_free is also used

Question

I'm writing a small program to test an old issue on RSA. I need to access the prime factors of the modulus. So my code is

int RSAKeyGen(int keySize) {

      EVP_PKEY *pkey  = EVP_PKEY_new();

      BIGNUM *bn = BN_new();
      BN_set_word(bn, RSA_F4);

      RSA *rsa = RSA_new();
      RSA_generate_key_ex(rsa, keySize, bn, NULL);

      EVP_PKEY_assign_RSA(pkey, rsa);

      const BIGNUM *p;
      const BIGNUM *q;

      RSA_get0_factors(rsa, &p,&q);

      BN_print_fp(stdout, p);
      puts("\n");
      BN_print_fp(stdout, q);

      RSA_free(rsa);
      EVP_PKEY_free(pkey);
      BN_free(bn);

      return 0;
}

Having both EVP_PKEY_free(pkey); and RSA_free(rsa); gives

`Segmentation fault (core dumped)`

commenting one of them is working fine, except may be something is not freed. Changed the order but did not worked.

• Does only one of them is enough to work correctly?

Answer 1

See the documentation for EVP_PKEY_assign_RSA here:

https://www.openssl.org/docs/man1.1.1/man3/EVP_PKEY_assign_RSA.html

As stated on that page:

EVP_PKEY_assign_RSA(), EVP_PKEY_assign_DSA(), EVP_PKEY_assign_DH(), EVP_PKEY_assign_EC_KEY(), EVP_PKEY_assign_POLY1305() and EVP_PKEY_assign_SIPHASH() also set the referenced key to key however these use the supplied key internally and so key will be freed when the parent pkey is freed.

So in other words ownership of the key assigned via the EVP_PKEY_assign_RSA() call is transferred to the EVP_PKEY. When you free the EVP_PKEY it also frees the underlying RSA key. So, once you've successfully called EVP_PKEY_assign_RSA() you must not call RSA_free() on the underlying key or a double-free may result.