Verify public key matches signature of private key

Question

I'm new to openssl API.

my goal is to verify that my public key is "related" to my hidden private key.

solution using openssl cli:

I have a certificate , the private key is hidden (in HSM) I have a buffer:

echo "hello world!!!!" > sign.txt

using following commands I create sha256 signature of my buffer similar to what my HSM will do :

openssl dgst -sha256 -sign myrootca.key.insecure -out sign.sha256 sign.txt

extract public key:

openssl x509 -pubkey -noout -in myrootca.crt  > myrootca.publicKey.pem

verify public key :

openssl dgst -sha256 -verify myrootca.publicKey.pem -signature sign.sha256 sign.txt

---

I think I know how to represent my public key and signature file correctly

sigkey = load_pubkey(bio_err, keyfile, keyform, 0, NULL,
                                 e, "key file");
sigbio = BIO_new_file(sigfile, "rb");

but can't find a proper API to continue from here

Answer 1

Issue I've encountered was figuring out what is the signature length before hand to allocate place for the signature that must have prior knowledge on RSA key size and type of hash

sha256 + RSA1024 ~ 128byte
sha256 + RSA2048 ~ 256byte

using openssl API based of this guide I've managed to verify signature using following code :

Verifyx509VsPrivKeySig(X509* x509Cert,char* signature,size_t sigLen,char* message,size_t messageLen)
{
        int rc;
        EVP_PKEY* pPubkey = NULL;
        EVP_MD_CTX* ctx = NULL;

        rc = getX509Publickey(x509Cert, &pPubkey);
        if (rc != SSL_OK)
        {
                goto err_verify;
        }

        ctx = EVP_MD_CTX_create();
        if(ctx == NULL) {
                PRINT_LOG_ERR("EVP_MD_CTX_create failed, error 0x%lx\n", ERR_get_error());
                /* failed */
                goto err_verify;
        }

        rc = EVP_DigestVerifyInit(ctx, NULL, EVP_sha256(), NULL, pPubkey);
        if(rc != 1) {
                PRINT_LOG_ERR("EVP_DigestVerifyInit failed, error 0x%lx\n", ERR_get_error());
                goto err_EVP_XTX_destroy; /* failed */
        }

        rc = EVP_DigestVerifyUpdate(ctx, message, messageLen);
        if(rc != 1) {
                PRINT_LOG_ERR("EVP_DigestVerifyUpdate failed, error 0x%lx\n", ERR_get_error());
                goto err_EVP_XTX_destroy; /* failed */
        }

        /* Clear any errors for the call below */
        ERR_clear_error();

        rc = EVP_DigestVerifyFinal(ctx,(const unsigned char*) signature, sigLen);
        if(rc != 1) {
                PRINT_LOG_ERR("EVP_DigestVerifyFinal failed, error 0x%lx\n", ERR_get_error());
                goto err_EVP_XTX_destroy; /* failed */
        }


        /*if we got to here , verify sig finished with good result!!*/
    if(ctx) {
        EVP_MD_CTX_destroy(ctx);
        ctx = NULL;
    }

    return true;
    //////////////////// ERROR HANDLING ///////////////////////////
err_EVP_XTX_destroy:
        EVP_MD_CTX_destroy(ctx);
err_verify:
        return false;
}