Decrypt OpenPGP Session Key

Question

I have a file encrypted using GPG that I would like to extract the session key from so I can decrypt the session key separately. I'm using Bouncy Castle to extract the session key using the following code:

private static void outputSessionKey(String path) throws FileNotFoundException, IOException {
    BCPGInputStream input = new BCPGInputStream(PGPUtil.getDecoderStream(new FileInputStream(path)));
    Packet packet;
    while((packet = input.readPacket()) != null) {
        if (packet instanceof PublicKeyEncSessionPacket) {
            PublicKeyEncSessionPacket encPacket = (PublicKeyEncSessionPacket) packet;
            byte[] encKey = encPacket.getEncSessionKey()[0];
            FileOutputStream output = new FileOutputStream("session_key_enc.bin");
            output.write(encKey);
            output.close();
         }
     }

    input.close();
}

I then decrypt the session key using the following OpenSSL command:

openssl rsautl -decrypt -in session_key_enc.bin -out session_key_dec.bin -inkey private.pem -raw

I would expect the raw output to now be the decrypted session key but I am unable to use it to decrypt the original file using --override-session-key.

Does anyone have an idea about what could be going wrong with this setup?

Answer 1

I see two things (at least) that should go wrong.

(0) You are clearly assuming the key (and algorithm) used is RSA, which is not the only encryption algorithm used by PGP and GPG.

(1) The element(s) of PublicKeyEncSessionPacket.getEncSessionKey() is(are) the results of MPInteger.getEncoded() i.e. the MPI encoding from rfc4880 sec 3.2 which is a two-octet bit length followed by the actual value octets. This value when written should be too large for RSA decryption and rsautl -decrypt should have given an error, at least assuming you are using the correct key.

(2) PGP's RSA encryption uses PKCS1-v1_5 (type 02) padding. After correcting (1) you need to use the default -pkcs not -raw.

With those changes it works for me. Note the RSA-encrypted key block 'm', as per sec 5.1 consists of 1 octet symmetric-algorithm, the actual key, and 2 octets checksum; the first two are separated by a colon and the third is not included in GPG's session-key.