RuntimeException when closing CipherInputStream

Question

I kind of got stuck with this exception:

java.lang.RuntimeException: error:0407806d:RSA routines:decrypt:DATA_LEN_NOT_EQUAL_TO_MOD_LEN
 at com.android.org.conscrypt.NativeCrypto.RSA_private_decrypt(Native Method)
 at com.android.org.conscrypt.OpenSSLCipherRSA.engineDoFinal(OpenSSLCipherRSA.java:274)
 at javax.crypto.Cipher.doFinal(Cipher.java:1440)
 at javax.crypto.CipherInputStream.close(CipherInputStream.java:190)
 ...

It is thrown when I close the CipherInputStream on Android Marshmallow. Everything seems to work with earlier Android Versions.

What does DATA_LEN_NOT_EQUAL_TO_MOD_LEN mean? Why does it seem to decrypt (call to RSA_private_decrypt) when it should free resource handles (close)?

UPDATE:

I managed to reproduce the error with some test code. It encrypts and decrypts "foobar". One time using the cipher directly and one time through a CipherInputStream (like it's done in the original app).

Everything works on android < 6 and the non-streaming code is even working on android 6. I was able to get the streaming code to work on android 6 when I changed the explicit cipher RSA/ECB/PKCS1Padding to generic RSA. But I would bet that it's there for a reason ;)

static final String RSA_ALGO = "RSA/ECB/PKCS1Padding";
//  static final String RSA_ALGO = "RSA";

private void _testCrypto2() throws Exception {
  KeyPairGenerator keyGen;
  KeyPair          keys;
  byte[]           encrypted;
  byte[]           decrypted;
  String           input;
  String           output;

  keyGen = KeyPairGenerator.getInstance("RSA");
  keyGen.initialize(2048);
  keys = keyGen.generateKeyPair();

  input = "foobar";

  // Plain crypto.
  encrypted = this.RSAEncrypt(input, keys.getPublic());
  output    = this.RSADecrypt(encrypted, keys.getPrivate());

  // Streaming crypto.
  encrypted = this.RSAEncryptStream(input, keys.getPublic());
  output    = this.RSADecryptStream(encrypted, keys.getPrivate());
}

public byte[] RSAEncrypt(final String plain, PublicKey _publicKey) throws Exception {
  byte[] encryptedBytes;
  Cipher cipher;

  cipher = Cipher.getInstance(RSA_ALGO);
  cipher.init(Cipher.ENCRYPT_MODE, _publicKey);
  encryptedBytes = cipher.doFinal(plain.getBytes());

  return encryptedBytes;
}

public String RSADecrypt(final byte[] encryptedBytes, PrivateKey _privateKey) throws Exception {
  Cipher cipher;
  byte[] decryptedBytes;
  String decrypted;

  cipher = Cipher.getInstance(RSA_ALGO);
  cipher.init(Cipher.DECRYPT_MODE, _privateKey);

  decryptedBytes = cipher.doFinal(encryptedBytes);
  decrypted      = new String(decryptedBytes);

  return decrypted;
}

public byte[] RSAEncryptStream(final String _plain, PublicKey _publicKey) throws Exception {
  Cipher                cipher;
  InputStream           in;
  ByteArrayOutputStream out;
  int                   numBytes;
  byte                  buffer[] = new byte[0xffff];

  in     = new ByteArrayInputStream(_plain.getBytes());
  out    = new ByteArrayOutputStream();
  cipher = Cipher.getInstance(RSA_ALGO);
  cipher.init(Cipher.ENCRYPT_MODE, _publicKey);

  try {
    in = new CipherInputStream(in, cipher);
    while ((numBytes = in.read(buffer)) != -1) {
      out.write(buffer, 0, numBytes);
    }
  }
  finally {
    in.close();
  }

  return out.toByteArray();
}

public String RSADecryptStream(final byte[] _encryptedBytes, PrivateKey _privateKey) throws Exception {
  Cipher                cipher;
  InputStream           in;
  ByteArrayOutputStream out;
  int                   numBytes;
  byte                  buffer[] = new byte[0xffff];

  in     = new ByteArrayInputStream(_encryptedBytes);
  out    = new ByteArrayOutputStream();  
  cipher = Cipher.getInstance(RSA_ALGO);
  cipher.init(Cipher.DECRYPT_MODE, _privateKey);

  try {
    in = new CipherInputStream(in, cipher);
    while ((numBytes = in.read(buffer)) != -1) {
      out.write(buffer, 0, numBytes);
    }
  }
  finally {
    in.close();
  }

  return new String(out.toByteArray());
}

However, it looks like there are two directions for a fix:

• Getting rid of the streaming for RSA
• Removing explicit RSA cipher instantiation

What do you think?

Answer 1

It looks like there were some changes for the default security providers of android.

Cipher        c;
Provider      p;
StringBuilder bldr;

c    = Cipher.getInstance("RSA");
p    = cipher.getProvider();
bldr = new StringBuilder();

bldr.append(_p.getName())
  .append(" ").append(_p.getVersion())
  .append(" (").append(_p.getInfo()).append(")");
Log.i("test", bldr.toString());

It seems to use a version of BouncyCastle on all tested Android versions (I tested down to 2.3):

• Android 5: BC 1.5 (BouncyCastle Security Provider v1.50)
• Android 6: BC 1.52 (BouncyCastle Security Provider v1.52)

However, something changed with the "explicit" cipher:

c = Cipher.getInstance("RSA/ECB/PKCS1Padding");

• Android 4.1.2: BC 1.46 (BouncyCastle Security Provider v1.46)
• Android 4.4.2: AndroidOpenSSL 1.0 (Android's OpenSSL-backed security provider)
• Android 5.1.1: AndroidOpenSSL 1.0 (Android's OpenSSL-backed security provider)
• Android 6.0.1: AndroidKeyStoreBCWorkaround 1.0 (Android KeyStore security provider to work around Bouncy Castle)

So the final solution is setting the provider explicitly to BouncyCastle which is working on all tested android versions, even with streaming:

Provider p;
Cipher   c;

p = Security.getProvider("BC");
c = Cipher.getInstance("RSA/ECB/PKCS1Padding", p);