I am a beginner who is trying to understand the basics of OpenSSL 3.0 by browsing the codebase. I started with exploring how RSA signing works by using the "openssl pkeyutl" app.
So if I understand the OpenSSL 3.0 design correctly, there is a provider which does the actual cryptographic functions. The provider could be the default provider which is part of the OpenSSL package or could be even a 3rd party library which has support for hardware acceleration.
Based on the documentation here, the signature init function needs to be called prior to signing.
OSSL_FUNC_signature_sign_init() initialises a context for signing given a provider side signature context in the ctx parameter, and a pointer to a provider key object in the provkey parameter. The params, if not NULL, should be set on the context in a manner similar to using OSSL_FUNC_signature_set_ctx_params(). The key object should have been previously generated, loaded or imported into the provider using the key management (OSSL_OP_KEYMGMT) operation (see provider-keymgmt(7)>.
int OSSL_FUNC_signature_sign_init(void *ctx, void *provkey,
const OSSL_PARAM params[]);
For the default provider, the OSSL_FUNC_signature_sign_init() calls the rsa_signverify_init() in rsa_sig.c.
static int rsa_signverify_init(void *vprsactx, void *vrsa,
const OSSL_PARAM params[], int operation)
{
PROV_RSA_CTX *prsactx = (PROV_RSA_CTX *)vprsactx;
if (!ossl_prov_is_running())
return 0;
if (prsactx == NULL || vrsa == NULL)
return 0;
if (!ossl_rsa_check_key(prsactx->libctx, vrsa, operation))
return 0;
if (!RSA_up_ref(vrsa))
return 0;
RSA_free(prsactx->rsa);
prsactx->rsa = vrsa;
prsactx->operation = operation;
In the above function the prsactx->rsa is set to vrsa which is the provkey passed by the upper layer. The rsa structure contains the list of function pointers that perform the actual RSA related cryptographic functions.
I am confused why the upper layers are supplying the functions to the provider? Isn't implementing these functions the entire job of provider?
The "upper layers" are just passing a "void *" handle for the key - which is obtained from the provider's own key management functions. The provider itself is responsible for constructing the key object.
In the case of RSA in the default provider you can see the code to do that here:
https://github.com/openssl/openssl/blob/master/providers/implementations/keymgmt/rsa_kmgmt.c
The provider key management capabilities are documented here:
https://www.openssl.org/docs/man3.0/man7/provider-keymgmt.html