Why do encryped RSA bytes in Java variate?

Question

I try to code a hybrid encryption to communicate between server and client. So I send a public RSA key from the client the server enrypts his AES key and sends it back. But then if i decrypt it on client site, the key is longer than i send it and i don't know why...

Here the code:

Client:

socket.getOutputStream().write(security.getKeyPair().getPublic().getEncoded());
byte[] keyBuffer = new byte[512];
socket.getInputStream().read(keyBuffer);
security.setKey(new SecretKeySpec(security.decryptRSA(keyBuffer).getBytes(), "AES"));

Server:

byte[] keyBuffer = new byte[550];
this.socket.getInputStream().read(keyBuffer);
PublicKey publicKey = KeyFactory.getInstance("RSA").generatePublic(new X509EncodedKeySpec(keyBuffer));
this.socket.getOutputStream().write(this.security.encryptRSA(this.security.getKey().getEncoded(), publicKey));

Security class methods:

public byte[] encryptRSA(byte[] message, PublicKey key) {
        byte[] buffer = message;
        try {
            this.cipher = Cipher.getInstance("RSA");
            this.cipher.init(Cipher.ENCRYPT_MODE, key);
            buffer = this.cipher.doFinal(buffer);
        } catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
            e.printStackTrace();
        }

        return buffer;
    }


public String decryptRSA(byte[] message) {
        byte[] buffer = message;
        try {
            this.cipher = Cipher.getInstance("RSA");
            this.cipher.init(Cipher.DECRYPT_MODE, this.keyPair.getPrivate());
            buffer = this.cipher.doFinal(buffer);
        } catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | IllegalBlockSizeException | BadPaddingException e) {
            e.printStackTrace();
        }

        return new String(buffer);
    }

Thanks!

Answer 1

The problem is likely that you stringified your code needlessly:

   return new String(buffer);

This will interpret the buffer in a certain way as a string. However, the key consists of random bytes, which may decode to unexpected characters. Then you revert back to bytes using .getBytes(), but by then it is too late.

From the Java String(byte[]): String constructor documentation:

The behavior of this constructor when the given bytes are not valid in the default charset is unspecified. The CharsetDecoder class should be used when more control over the decoding process is required.

Usually though it simply silently drops bytes that cannot be decoded, although representing the unknown encoding with a Unicode character or question mark may also happen. In any of these operations information is lost.

Simply leave the AES key to be bytes and that part should be fixed.

---

Besides that, read is not the same as readNBytes​ (and just read is wrong), but that's probably not the error; you'd get into trouble with RSA, not the AES key if it would be the problem.

---

Note that you can retrieve the modulus size in bytes to determine the amount of bytes to read, so you don't need to have a 512 constant there either (which I would write as KEY_SIZE / Byte.SIZE where KEY_SIZE = 4096 to at least indicate that the buffer is the same as the key size clearly.