wc_RsaSSL_Verify returns BAD_FUNC_ARG, and I can't tell why

Question

I am trying to RSA public key decrypt a signed file using wolfcrypt - yes, I may or may not be abusing the "sign/verify" power of RSA to encrypt a separate AES key using the private key and decrypt using the public key.

Unfortunately, I am stuck at wc_RsaSSL_Verify() - for the life of me I can't figure out why it is returning BAD_FUNC_ARG - I figured an error like that should be immediately visible to somebody else so I'm deciding to call upon the collective powers of StackOverflow.

As far as I can tell, I'm giving the function what it's asking for - an input buffer, an output buffer, the size of each, and a pointer to the RsaKey struct. Here is a code snippet from the function in question:

bool VerifyWorker::GetAESKey()
{
  bool result = true;
  uint8_t en_aes_file_buff[VerifyWorkerLocal::RSA_KEY_SIZE];
  uint8_t de_aes_file_buff[VerifyWorkerLocal::RSA_KEY_SIZE];

  uint8_t* aes_iv_ptr = NULL;

  // keyfile filestream
  std::fstream aes_file;

  // rsa_key must be initialized
  if(rsa_key == NULL)
  {
     result = false;
  }

  // Open the key file and read it into a local buffer, then decrypt it and use it to initialize the
  // aes struct
  if(result)
  {
     aes_file.open(this->aes_key_file, std::ios_base::in | std::ios_base::binary);

     if(aes_file.fail())
     {
        // Unable to open file - perror?
        perror("GetAESKey");
        result = false;
     }
     else
     {
        aes_file.read(reinterpret_cast<char*>(en_aes_file_buff), VerifyWorkerLocal::RSA_KEY_SIZE + 1);
        if(!aes_file.eof())
        {
           // we didn't have enough space to read the whole signature!
           std::cerr << "aes_file read failed! " << aes_file.rdstate() << std::endl;
           result = false;
        }
     }
  }

  // "Unsign" the aes key file with RSA verify, and load the aes struct with the result
  if(result)
  {
     int wc_ret = 0;
     wc_ret = wc_RsaSSL_Verify(const_cast<const byte*>(en_aes_file_buff),
                               VerifyWorkerLocal::RSA_KEY_SIZE, reinterpret_cast<byte*>(&de_aes_file_buff),
                               VerifyWorkerLocal::RSA_KEY_SIZE, rsa_key);

The rsa_key is a private member initialized (successfully, using wc_PublicKeyDecode()) in a separate function with a public key DER file. I generated both the public and private key using OpenSSL - which should properly pad my AES key and iv file using PKCS#1 v1.5 b default.

I should also mention that I am using wolfssl version 3.9.8. Thanks!

Answer 1

The issue, I found, was that the file that I had signed with my RSA key was not signed correctly. When I signed the file using OpenSSL, my cli invocation was

openssl rsautl -in keyfile -out keyfile -inkey private.pem -sign

Apparently, openssl does not like you to specify the same file for -in and -out. When I changed it to something like

openssl rsautl -in keyfile -out keyfile_signed -inkey private.pem -sign

I was actually able to verify the file using wc_RsaSSL_Verify.

So, like most stupid late-night, last hour software problems, I was looking in the wrong place entirely. I was a bit thrown off by the BAD_FUNC_ARG being returned and thought that it had to do explicitly with the format of the function arguments, not necessarily their content. Hopefully this answer is useful for somebody else, too.