verifying digest signed with private key using a C program

Question

I'll explain all the steps I've done so far and conclude with my question.

using OpenSSL 1.0.1e-fips 11 Feb 2013

Generating a private and public key

openssl genrsa -des3 -out private.pem 2048
openssl rsa -in private.pem -outform PEM -pubout -out public.pem

signing a message with digest and private key

openssl dgst -sha256 -sign private.pem -out message.secret message.txt

at this point I have a public key, a signed message ( with digest ) and the original message.

Part 1 - using CLI ( this one works )

Using the CLI I manage to verify the digest:

openssl dgst -sha256 -verify public.pem -signature message.secret message.txt

I get "Verified OK" as a return value.

Part 2 - Using C program

My program looks like this:

where:

msg is message.txt

signature is message.secret

pkey is the public key ( achieved using PEM_read_PUBKEY )

int verify_it(const byte* msg, size_t msg_len, byte* signature, EVP_PKEY* pkey) {
    EVP_MD_CTX      *ctx;
    size_t          sig_len;
    int             bool_ret;
    int             ret_val;

    ret_val = EXIT_SUCCESS;
    ctx = NULL;

    do
    {
        ctx = EVP_MD_CTX_create();    
        const EVP_MD* md = EVP_get_digestbyname( "SHA256" );

        EVP_DigestInit_ex( ctx, md, NULL );
        EVP_DigestVerifyInit( ctx, NULL, md, NULL, pkey );
        EVP_DigestVerifyUpdate(ctx, msg, msg_len);

        sig_len = 256;            
        if ( !EVP_DigestVerifyFinal( ctx, signature, sig_len ) 
                ERR_print_errors_fp( stdout )
        );

    } while(0);

    return ret_val;

}

this code returns a verification failure ( value of 0 ).

also the function ERR_print_errors_fp( stdout ) prints the following message:

140332412258152:error:04091077:lib(4):func(145):reason(119):rsa_sign.c:176

EDIT

thanks to @deniss I've managed to fix this one problem ( signature length was bad - probably a '\0' mid signature - so i just edit the length to be 256 )

but now I'm getting another problem -

140195987986280:error:04091068:lib(4):func(145):reason(104):rsa_sign.c:293

checking it with openssl errstr I've got

error:04091068:rsa routines:INT_RSA_VERIFY:bad signature

the way I acquire my signature is like this:

secret_fp = fopen( "message.secret", "rb" );
fseek( secret_fp, 0, SEEK_END );
file_len = ftell( secret_fp );
fseek( secret_fp, 0, SEEK_SET );

signature = malloc( file_len );
fread( signature, file_len, 1, secret_fp );

Answer 1

You can always decode openssl error codes into meaningful messages with

openssl errstr <error-code>

Code 04091077 stands for error:04091077:rsa routines:INT_RSA_VERIFY:wrong signature length.

The most probable explanation:

• message.secret is binary file

• it has zero byte somewhere in the middle

• strlen trims signature on this byte