encryption with java RSA and decryption with BigInteger

Question

So I am trying to figure out how I can encrypt and decrypt RSA in java. Encryption is done with javas API and decryption with biginteger. I am finished but biginteger is giving me weird output sometimes. This is my code:

import java.math.BigInteger;
import java.security.InvalidKeyException;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.NoSuchAlgorithmException;
import java.security.NoSuchProviderException;
import java.security.Security;
import java.security.interfaces.RSAPrivateKey;
import java.security.interfaces.RSAPublicKey;
import java.security.spec.InvalidKeySpecException;

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;

import org.bouncycastle.jce.provider.BouncyCastleProvider;

public class cryptoAndBigIntegerFIX {
    public static void main(String[] args) throws NoSuchAlgorithmException, NoSuchProviderException, InvalidKeyException, NoSuchPaddingException, IllegalBlockSizeException, BadPaddingException, InvalidKeySpecException{
        Security.addProvider(new BouncyCastleProvider());
        System.out.println("input <> encrypted <> decrypted");
        cryptoAndBigIntegerFIX.keygen();

        BigInteger encryptbytes;
        BigInteger decryptbytes;

        //Multiple tests with powers of 3 for some reason :D
        for(int i=1;i<1000;i*=3){
            encryptbytes = cryptoAndBigIntegerFIX.encrypt(new BigInteger(""+i));
            System.out.print(i + " <> " + encryptbytes.intValue() + " <> ");
            decryptbytes = cryptoAndBigIntegerFIX.decrypt(encryptbytes);
            System.out.println(decryptbytes.intValue());            
        }
    }

    public static RSAPrivateKey priv;
    public static RSAPublicKey pub;

    public static void keygen() throws NoSuchAlgorithmException, NoSuchProviderException, InvalidKeySpecException{
        KeyPairGenerator generator = KeyPairGenerator.getInstance("RSA");

        generator.initialize(512);
        KeyPair keyPair = generator.generateKeyPair();
        priv = (RSAPrivateKey) keyPair.getPrivate();
        pub = (RSAPublicKey) keyPair.getPublic();       
    }

    //Encrypt with javas API
    public static BigInteger encrypt(BigInteger bg) throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException{
        byte[] encoded;
        Cipher cipher=Cipher.getInstance("RSA/ECB/NoPadding");
        cipher.init(Cipher.ENCRYPT_MODE, pub);
        encoded=cipher.doFinal(bg.toByteArray());
        return new BigInteger(encoded);
    }

    //Decrypt manually
    public static BigInteger decrypt(BigInteger bg){
        BigInteger decoded = bg.modPow(priv.getPrivateExponent(),priv.getModulus());
        return decoded;

    }

}

The output this gives is:

input <> encrypted <> decrypted
1 <> 1 <> 1
3 <> 1088098617 <> 3
9 <> 1947497039 <> 9
27 <> -1665331145 <> 27
81 <> -1064046970 <> 81
243 <> -599005266 <> 243
729 <> -1534949160 <> 729

which is correct as I am trying to encrypt and decrypt powers of three. But sometimes it gives the wrong output like:

input <> encrypted <> decrypted
1 <> 1 <> 1
3 <> 1693488667 <> 3
9 <> -924345856 <> 9
27 <> 777525903 <> 144224668
81 <> -1602799071 <> 765474161
243 <> -227258229 <> 243
729 <> 1097077312 <> 296615835

which is pretty weird? Any idea what is wrong with my code? Is it an issue with biginteger or with the way i generate keys?

Answer 1

There are two mistakes in your code, fixing these results in correct output:

• RSA returns a byte array that represents an encoding of an unsigned (big endian) value, while the BigInteger constructor expects a signed (big endian) value;
• you convert your ~512 bit integers to a 32 bit int using intValue, which is a narrowing conversion (i.e. you lose most of the information);

---

To fix this use the following code in your main:

for (int i = 1; i < 1000; i *= 3) {
    encryptbytes = encrypt(new BigInteger("" + i));
    decryptbytes = decrypt(encryptbytes);
    System.out.printf("%d <> %d <> %d%n", i, encryptbytes, decryptbytes); // <-- CHANGED
}

and fix your encrypt method like this:

public static BigInteger encrypt(BigInteger bg) throws NoSuchAlgorithmException, NoSuchPaddingException, InvalidKeyException, IllegalBlockSizeException, BadPaddingException, NoSuchProviderException{
    byte[] encoded;
    Cipher cipher=Cipher.getInstance("RSA/ECB/NoPadding");
    cipher.init(Cipher.ENCRYPT_MODE, pub);
    encoded=cipher.doFinal(bg.toByteArray());
    // always return a positive number, all numbers in RSA range from 0 to N, the modulus
    return new BigInteger(1, encoded); // <-- CHANGED
}

---

Output:

1 <> 1 <> 1
3 <> 5391244976364305547335484748020521942086098609617330384518129302949818422340576199311962549547267600132422679070440104879858341377467952359964277978034437 <> 3
9 <> 6124821216540311592072512276604516925109106437800474377305024835746311326643844937438898166014089888517013623381274703706957136996030394061334489396421471 <> 9
27 <> 7417955231776825208837885603385850571492475371613109504900294367112291012186031184819506881671785651668774452694772296927652149185293472327267855826108811 <> 27
81 <> 4379633872520787556108807271103986910459839084415924804306544674047140392288351081810043910720852387318488882506988222018682090612584816530149697340650488 <> 81
243 <> 7246933972891920591130671309726014055879369415307203680175037003794552795366338573568427805702031164261947749468386326443821678079330776559688467025349997 <> 243
729 <> 1545447812776244759821683256943744637177323415383752406121617316817182282617298911713206245407992744746920675542053503316837741108184755354755351275313816 <> 729

---

Warning for other readers: never use plaintext/raw RSA ("RSA/ECB/NoPadding" in Java), it is insecure. Try a hybrid cryptosystem with RSA OAEP instead. Do not copy/paste above code.