Bouncy Castle Extract PGP Session Key from Public-Key Encrypted Session Packet

Question

I have a PGP Public-Key Encrypted Session Packet from which I would like to extract the session key so that I can decrypt the session key separately. I'm using the BouncyCastle library and am extracting the session key like this:

private static void outputSessionKey(String path) throws FileNotFoundException, IOException {
    BCPGInputStream input = new BCPGInputStream(PGPUtil.getDecoderStream(new FileInputStream(path)));
    Packet packet;
    while((packet = input.readPacket()) != null) {
        if (packet instanceof PublicKeyEncSessionPacket) {
            PublicKeyEncSessionPacket encPacket = (PublicKeyEncSessionPacket) packet;
            byte[] encKey = encPacket.getEncSessionKey()[0];
            FileOutputStream output = new FileOutputStream("session_key_enc.bin");
            output.write(encKey);
            output.close();
        }
    }
    
    input.close();
}

I was expecting to then be able to decrypt the session key using openssl:

openssl rsautl -decrypt -in session_key_enc.bin -out session_key_decoded.bin -inkey private.pem

Where session_key_enc.bin is my encrypted session key in binary format and private.pem is the corresponding private key to the public key I used to encrypt my data in GPG. Before encrypting my data, I converted the public key portion of my RSA key pair into a PGP formatted key and imported it into GPG.

When I run the OpenSSL command, I get this error:

RSA operation error
140624851898072:error:0406506C:rsa routines:RSA_EAY_PRIVATE_DECRYPT:data greater than mod len:rsa_eay.c:518:

Upon examining session_key_enc.bin I found that the file is 258 bytes. This doesn't seem like it should be possible considering I'm using a 2048 bit RSA key and the spec indicates that the encrypted session key is modded by n:

Algorithm Specific Fields for RSA encryption - multiprecision integer (MPI) of RSA encrypted value m**e mod n.

The value "m" in the above formulas is derived from the session key as follows. First, the session key is prefixed with a one-octet algorithm identifier that specifies the symmetric encryption algorithm used to encrypt the following Symmetrically Encrypted Data Packet. Then a two-octet checksum is appended, which is equal to the sum of the preceding session key octets, not including the algorithm identifier, modulo 65536. This value is then encoded as described in PKCS#1 block encoding EME-PKCS1-v1_5 in Section 7.2.1 of [RFC3447] to form the "m" value used in the formulas above. See Section 13.1 of this document for notes on OpenPGP's use of PKCS#1.

Any advice on how to solve this puzzle would be much appreciated, thanks!

Answer 1

Turns out Bouncy Castle exports encrypted session keys using MPI format where the first 2 bytes are the length. This resolves the original issue where I was unable to decrypt the session key because it was 258 bytes instead of 256.

I am marking this question as answered despite still being unable to decrypt the file using --override-session-key and the raw bytes of the now-decrypted session key.