Search code examples
androidhashrsaandroid-keystore

android.security.KeyStoreException: Incompatible digest when signing with RSA


When I try to sign a hashed value in my android app that I get from outside, I get the above mentioned exception.

The code for generating the keypair is:

public static KeyPair generateKeyPair(Context context, String username) throws Exception {
        KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance(KeyProperties.KEY_ALGORITHM_RSA, ANDROID_KEY_STORE);
        Calendar start = Calendar.getInstance();
        Calendar end = Calendar.getInstance();
        end.add(Calendar.YEAR, 1);
        AlgorithmParameterSpec parameterSpec;
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            parameterSpec = new KeyGenParameterSpec.Builder(MY_KEY_ALIAS, KeyProperties.PURPOSE_SIGN)
                    .setKeySize(KEY_SIZE)
                    .setCertificateSubject(usernameToSubject(username))
                    .setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA1)
                    .setCertificateNotBefore(start.getTime())
                    .setCertificateNotAfter(end.getTime())
                    .setSignaturePaddings(KeyProperties.SIGNATURE_PADDING_RSA_PKCS1)
                    .build();
        } else {
           // Here I build the keys for older versions. This is not part of my problem
        }
        keyPairGenerator.initialize(parameterSpec);
        return keyPairGenerator.generateKeyPair();
    }

Later on I sign the hash I get from outside:

 public static byte[] signHash(byte[] hashToSign) {
        try {
            KeyStore keyStore = KeyStore.getInstance(ANDROID_KEY_STORE);
            keyStore.load(null);
            KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry)keyStore.getEntry(MY_KEY_ALIAS, null);
            PrivateKey privateKey = keyEntry.getPrivateKey();

            DigestAlgorithmIdentifierFinder hashAlgorithmFinder = new DefaultDigestAlgorithmIdentifierFinder();
            AlgorithmIdentifier hashingAlgorithmIdentifier = hashAlgorithmFinder.find(KeyProperties.DIGEST_SHA256);
            DigestInfo digestInfo = new DigestInfo(hashingAlgorithmIdentifier, hashToSign);
            byte[] hashToEncrypt = digestInfo.getEncoded();
            Cipher cipher = Cipher.getInstance("RSA/ECB/Pkcs1Padding");
            cipher.init(Cipher.ENCRYPT_MODE, privateKey); // <= the exception is thrown here
            return cipher.doFinal(hashToEncrypt);
        } catch (Throwable e) {
            Log.e("KeyStoreWrapper", "Error while signing: ", e);
        }
        return "Could not sign the message.".getBytes(StandardCharsets.UTF_16LE);
    }

The error I get is:

java.security.InvalidKeyException: Keystore operation failed
 at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1004)
 at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1024)
 at android.security.keystore.KeyStoreCryptoOperationUtils.getInvalidKeyExceptionForInit(KeyStoreCryptoOperationUtils.java:53)
 at android.security.keystore.KeyStoreCryptoOperationUtils.getExceptionForCipherInit(KeyStoreCryptoOperationUtils.java:89)
 at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:263)
 at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:108)
 at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:612)
 at javax.crypto.Cipher.tryCombinations(Cipher.java:532)
 at javax.crypto.Cipher.getSpi(Cipher.java:437)
 at javax.crypto.Cipher.init(Cipher.java:815)
 at javax.crypto.Cipher.init(Cipher.java:774)
 at de.new_frontiers.m2fa.security.KeyStoreWrapper.signHash(KeyStoreWrapper.java:186)
 at de.new_frontiers.m2fa.bluetooth.BluetoothHandler.handleMessage(BluetoothHandler.java:93)
 at android.os.Handler.dispatchMessage(Handler.java:102)
 at android.os.Looper.loop(Looper.java:158)
 at android.app.ActivityThread.main(ActivityThread.java:7229)
 at java.lang.reflect.Method.invoke(Native Method)
 at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1230)
 at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1120)
Caused by: android.security.KeyStoreException: Incompatible digest
 at android.security.KeyStore.getKeyStoreException(KeyStore.java:944)
 at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1024) 
 at android.security.keystore.KeyStoreCryptoOperationUtils.getInvalidKeyExceptionForInit(KeyStoreCryptoOperationUtils.java:53) 
 at android.security.keystore.KeyStoreCryptoOperationUtils.getExceptionForCipherInit(KeyStoreCryptoOperationUtils.java:89) 
 at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:263) 
 at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:108) 
 at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:612) 
 at javax.crypto.Cipher.tryCombinations(Cipher.java:532) 
 at javax.crypto.Cipher.getSpi(Cipher.java:437) 
 at javax.crypto.Cipher.init(Cipher.java:815) 
 at javax.crypto.Cipher.init(Cipher.java:774) 
 at com.mycompany.security.KeyStoreWrapper.signHash(KeyStoreWrapper.java:186) 

In the android documentation I see:

For signing and verification operations a digest must be specified in the additional_params argument of begin. If the specified digest is not in the digests associated with the key, the operation must fail with KM_ERROR_INCOMPATIBLE_DIGEST.

But as you can see I create the keypair with KeyProperties.DIGEST_SHA256 and set the DigestInfo to the same algorithm. So I don't understand why I get the error "Incompatible digest". Can anybody shed some light on this?

Oh, for anybody who is wondering why I don't use Signature.sign(): this would need the plaintext to sign then creates a hash, a DigestInfo and then encrypts it with the private key. I already get the hash from outside and this is something I cannot change.  


Solution

  • pedrofbs answer helped me to get things right in the end. I had already changed the purpose for the key to the value mentioned in my comment to his answer: KeyProperties.PURPOSE_SIGN|KeyProperties.PURPOSE_ENCRYPT|Key‌​Properties.PURPOSE_D‌​ECRYPT, but forgot to call .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_PKCS1). So a big thank you, pedrofs, for spotting this!

    Unfortunately it still did not work. Fiddling around with different settings on the key I realized that I hadn't protected the device I use for testing with a password, pin or something else. So adding .setUserAuthenticationRequired(false) to the KeyGenParameterSpec did the trick. I know that this is not safe and has to be changed but at the moment I'm just doing a proof of concept, so that's fine. :-)

    In case somebody else stumbles upon this problem: here is the whole working code for the key generation:

     if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
                parameterSpec = new KeyGenParameterSpec.Builder(LOGON_KEY_ALIAS, KeyProperties.PURPOSE_SIGN|KeyProperties.PURPOSE_ENCRYPT|KeyProperties.PURPOSE_DECRYPT)
                        .setKeySize(KEY_SIZE)
                        .setUserAuthenticationRequired(false)
                        .setCertificateSubject(usernameToSubject(username))
                        .setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA1)//, KeyProperties.DIGEST_NONE, KeyProperties.DIGEST_SHA224, KeyProperties.DIGEST_SHA384, KeyProperties.DIGEST_SHA512, KeyProperties.DIGEST_MD5)
                        .setCertificateNotBefore(start.getTime())
                        .setCertificateNotAfter(end.getTime())
                        .setSignaturePaddings(KeyProperties.SIGNATURE_PADDING_RSA_PKCS1)
                        .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_PKCS1)
                        .build();
            }