'dgst verify and sign' equivalent with 'RSA_Verify()'

I am using the RSA_verify() function to validate a SHA that I signed by using the openssl program (via a console). RSA_verify() is always returning a non successful validation, so I think that I am sending incorrect parameters to it.

The following console commands are run in Linux Ubuntu with OpenSSL 0.9.8k.

The C functions are compiled for Android, using OpenSSL 1.0.1...c as far as I remember. It's definitely 1.0.1 (we are updating it to avoid the Heartbleed issue).

This is what I do... please forgive any mistake as I am learning this by myself.

Generate a private key
```
openssl genrsa -out private.key 2048
```

Extract the public key from the private key

openssl rsa -in private.key -out public.key -outform PEM -pubout

Calculate the SHA from a file called permissions and then sign it with the private.key, the output will be a SHA but with RSA encryption (permissions.sign)
```
openssl dgst -sha256 -sign private.key -out permissions.sign permissions
```
Validate a received SHA signature against the permissions file (it's successful in the Ubuntu console)
```
openssl dgst -sha256 -verify public.key -signature permissions.sign permissions
```
I copy the permissions file, the permissions.sign file and the public.key file to the file system in Android.
I verify that permissions.sign was created with permissions and with a matching private.key... all of this with my public.key (I don't have the private key in Android).

This is the C function.

#include <openssl/pem.h>
#include <openssl/rsa.h>
#include <openssl/sha.h>

...

/* Initialize the public key */
RSA *pub_key = RSA_new();

if(NULL == pub_key)
{
   ANDROID_LOGE("RSA_new failed");
   result = 0;
}
else
{
   FILE* fp = fopen(public_key, "r");

   if(NULL == fp)
   {
      ANDROID_LOGE_P("fopen [%s] failed", public_key);
      result = 0;
   }
   else
   {
      /* Read it from the passed path */
      if(PEM_read_RSA_PUBKEY(fp, &pub_key, NULL, NULL) == NULL)
      {
         ANDROID_LOGE_P("[%s] can't be read", public_key);
         result = 0;
         fclose(fp);
      }
      else
      {
         /* Verify the file and its SHA with the public key */
         int verified = RSA_verify(
             NID_sha256,
             file, /* message digest (message to validate) */
             file_size, /* message size */
             sign, /* signature (signed SHA) */
             sign_size, /* signature size */
             pub_key);
         ANDROID_LOGD("NID_sha256");

         if(verified)
         {
            result = 1;
            ANDROID_LOGD_P("[%s] is valid", file_to_verify);
         }
         else
         {
            ANDROID_LOGE_P("[%s] is NOT valid", file_to_verify);
         }

         fclose(fp);
      }

      RSA_free(pub_key);
   }
}

public_key is a path to public.key
PEM_read_RSA_PUBKEY succeeds
NID_sha256 is what I think I should use for the verification
file is a byte array with the contents of permissions
file_size is the array size of file
sign is a byte array with the contents of permissions.sign
sign_size is the array size of sign
RSA_verify() fails, it returns a 0

So the question is, is it correct to:

generate the keys with the commands I used,
sign the permissions file (which generates permissions.sign),
and then try to verify the files with PEM_read_RSA_PUBKEY() and RSA_verify() ?

Are the commands I used for the signing process equivalent to the C functions I used for the verification process?

Please let me know if more info is required or where I could study more about this.

Thanks!

EDIT: I added some error printing after calling RSA_Verify():

ANDROID_LOGE_P("openssl: %s", ERR_reason_error_string(ERR_get_error()));

It prints:

openssl: bad signature

Still investigating.

Solution

There were several missing steps in my reading process for the public key.

The correct commands to generate the private and public keys are as follows:

Generate a private key "openssl genrsa -out private.key 2048"

Extract the public key (DER certificate form) from the private key (needed by RSA_SHA_Verify()) "openssl req -outform DER -new -x509 -key private.key -out public.key -days 30000"

Generate a public key with no certificate info (only needed by "openssl dgst -sha1 -verify ...") "openssl x509 -inform DER -in public.key -pubkey -noout > public_no_cert.key"

Sign a file with the private key "openssl dgst -sha1 -sign private.key -out permissions.sign permissions"

Verify a file with the public key (no certificate info) "openssl dgst -sha1 -verify public_no_cert.key -signature permissions.sign permissions"

Please refer to the documentation on OpenSSL.org for details. I required a X509 DER certificate that was holding the public key to verify the signed SHA with RSA_verify().

An equivalency in command mode to RSA_verify() is:

openssl dgst -sha1 -verify public.key -signature permissions.sign permissions

For code source, please refer to this link: http://www.bmt-online.org/geekisms/RSA_verify

It does NOT compile at first sight, you have to tweak it. Call the functions in there as follows:

   result = sign_data(
         input_file_buffer,
         input_file_size,
         private_key_buffer,
         private_key_size,
         (void**)&signature,
         &signature_size);

   result = verify_data(
         input_file_buffer,
         input_file_size,
         signature_buffer,
         signature_size,
         public_key_buffer,
         public_key_size);

Everything has to be in RAM, pass them as pointers.

The signing function expects the pointer to a pointer (**) to save the signed SHA into it. You can later save it into a file.

Tested under Ubuntu openssl 0.9.8k.

If you see that something is missing, please let me know. Thanks for reading!

EDIT: Here's a link to the source code... http://migsantiago.com/index.php/tutoriales/32-firma-y-valida-archivos-con-openssl