I am trying to generate digital signatures on Windows (from XP SP3, but currently testing with Windows 7) with CryptoAPI that will be compatible with the following openssl commands:
openssl dgst -sha256 -sign <parameters> (for signing)
openssl dgst -sha256 -verify <parameters> (for validation)
I want to use a private key from the Windows "MY" keystore for signing.
I managed to sign files using the SHA1 digest algorithm by using the following CryptoAPI functions (omitting parameters for brevity):
CertOpenStore
CertFindCertificateInStore
CryptAcquireCertificatePrivateKey
CryptCreateHash (with CALG_SHA1)
CryptHashData
CryptSignHash
The generated signature is compatible with "openssl dgst -sha1 -verify" (once the byte order is reversed).
My problem is: when I try to use CALG_SHA_256 with CryptCreateHash, it fails with error 80090008 (NTE_BAD_ALGID). By googling around, I found that I needed to use a specific provider (PROV_RSA_AES) instead of the default one. Since I would have a provider handle, I would also need to replace CryptAcquireCertificatePrivateKey by CryptGetUserKey. So I modified my program to look like:
CryptAcquireContext (with PROV_RSA_AES)
CertOpenStore
CertFindCertificateInStore
CryptGetUserKey
CryptCreateHash (with CALG_SHA256)
CryptHashData
CryptSignHash
Unfortunately, this didn't work as expected: CryptGetUserKey failed with error 8009000D (NTE_NO_KEY). If I remove the CryptGetUserKey call, the program runs until CryptSignHash, which fails with error 80090016 (NTE_BAD_KEYSET). I know the keyset does exist and works fine, since I was able to use it to sign the SHA1 digest.
I tried acquiring the context again with information from the certificate context I got from CertFindCertificateInStore: the best I could do was a successful CryptGetUserKey call, but CryptSignHash would always fail with the same error.
The private key I am trying to use is 2048 bits long, but I don't expect it to be a problem since it works with the SHA1 digest. I am at a loss, so any suggestion would be very welcome!
The problem is most likely to be that certificates on Windows "know" in which provider their private keys are stored. When you import your cert it would put the key into a certain provider type (probably PROV_RSA_FULL), when you then later try to access the key via the certificate it will probably end up in the same provider type.
You probably need to open the associated context for the certificate (have a look at CertGetCertificateContextProperty with the CERT_KEY_PROV_HANDLE_PROP_ID option). With that handle yyou could try exporting the key from the original provider context and reimporting into a new PROV_RSA_AES one (assuming the key is exportable).