RSA public key decryption on OS X using SecTransform API (or other system API)

Question

I'm trying to replace my use of OpenSSL, which was long ago deprecated and has been removed from the 10.11 SDK with the Security Transform API. My use of OpenSSL is simply for license key verification. The problem I've run into is that license keys are generated (server side) using OpenSSL's rsa_private_encrypt() function, rather than the (probably more appropriate) rsa_sign(). In the current OpenSSL code, I verify them using rsa_public_decrypt() like so:

int decryptedSize = RSA_public_decrypt([signature length], [signature bytes], checkDigest, rsaKey, RSA_PKCS1_PADDING);
BOOL success = [[NSData dataWithBytes:checkDigest length:decryptedSize] isEqualToData:[digest sha1Hash]]) 

Unfortunately, I'm unable to replicate this using the SecTransform APIs. I have the following:

SecTransformRef decryptor = CFAutorelease(SecDecryptTransformCreate(pubKey, &error));
if (error) { showSecError(error); return NO; }
SecTransformSetAttribute(decryptor, kSecTransformInputAttributeName, (CFDataRef)signatureData, &error);
if (error) { showSecError(error); return NO; }

CFDataRef result = SecTransformExecute(decryptor, &error);
if (error) { showSecError(error); return NO; }
return CFEqual(result, (CFDataRef)[digest sha1Hash]);

The call to SecTransformExecute() fails with a CSSMERR_CSP_INVALID_KEY_CLASS error.

Am I missing something, or is there no equivalent to OpenSSL's RSA_public_decrypt() in Security.framework? Perhaps a SecVerifyTransform can be used (I have been unable to get this to work either, but then the same is true of OpenSSL's RSA_sign()). I am certainly willing to use another system API (e.g. CDSA/CSSM) if it will enable me to do this.

Unfortunately, as this code needs to verify existing license codes, I can not simply change my license generation code to use RSA_sign() or similar instead.

Answer 1

I figured out how to do this using CDSA/CSSM. Code below:

#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wdeprecated-declarations"

NSData *ORSDecryptDataWithPublicKey(NSData *dataToDecrypt, SecKeyRef publicKey)
{
    const CSSM_KEY *cssmPubKey = NULL;
    SecKeyGetCSSMKey(publicKey, &cssmPubKey);
    CSSM_CSP_HANDLE handle;
    SecKeyGetCSPHandle(publicKey, &handle);

    CSSM_DATA inputData = {
        .Data = (uint8_t *)[dataToDecrypt bytes],
        .Length = [dataToDecrypt length],
    };

    CSSM_DATA outputData = {
        .Data = NULL,
        .Length = 0,
    };

    CSSM_ACCESS_CREDENTIALS credentials;
    memset(&credentials, 0, sizeof(CSSM_ACCESS_CREDENTIALS));
    CSSM_CC_HANDLE contextHandle;
    CSSM_RETURN result = CSSM_CSP_CreateAsymmetricContext(handle, cssmPubKey->KeyHeader.AlgorithmId, &credentials, cssmPubKey, CSSM_PADDING_PKCS1, &contextHandle);
    if (result) { NSLog(@"Error creating CSSM context: %i", result); return nil; }

    CSSM_CONTEXT_ATTRIBUTE modeAttribute = {
        .AttributeType = CSSM_ATTRIBUTE_MODE,
        .AttributeLength = sizeof(UInt32),
        .Attribute.Uint32 = CSSM_ALGMODE_PUBLIC_KEY,
    };
    result = CSSM_UpdateContextAttributes(contextHandle, 1, &modeAttribute);
    if (result) { NSLog(@"Error setting CSSM context mode: %i", result); return nil; }

    CSSM_SIZE numBytesDecrypted = 0;
    CSSM_DATA remData = {
        .Data = NULL,
        .Length = 0,
    };
    result = CSSM_DecryptData(contextHandle, &inputData, 1, &outputData, 1, &numBytesDecrypted, &remData);
    if (result) { NSLog(@"Error decrypting data using CSSM: %i", result); return nil; }
    CSSM_DeleteContext(contextHandle);

    outputData.Length = numBytesDecrypted;

    return [NSData dataWithBytesNoCopy:outputData.Data length:outputData.Length freeWhenDone:YES];
}

#pragma clang diagnostic pop

Note that as documented here, while CDSA is deprecated, Apple recommends its use "if none of the other cryptographic service APIs support what you are trying to do". I have filed radar #23063471 asking for this functionality to be added to Security.framework.