BouncyCastle and SunJCE different result in Cipher::update and Cipher::doFinal

Question

I was trying to switch security provider from SunJCE to Bouncy Castle (BC) and stumbled upon this peculiar behaviour in the Cipher object. To my knowledge the encrypted text returned by SunJCE's cipher.update(bytes) includes the subsequent initialization vector (IV) as the last block. With BC, I need to call cipher.doFinal() and take the first block to get the IV. The algorithm I'm using is AES/CBC/PKCS5Padding

Why is this happening and what's the best way to handle this?

Here's my code

import org.bouncycastle.jce.provider.BouncyCastleProvider;

import javax.crypto.Cipher;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.security.Security;
import java.util.Arrays;
import java.util.Base64;

import static java.nio.charset.StandardCharsets.UTF_8;
import static javax.xml.bind.DatatypeConverter.printHexBinary;

public class CipherDebug {

    private final String algorithm;

    private final String provider;

    private final String cryptographicAlgorithm;

    public CipherDebug(String algorithm,
                       String cipherMode,
                       String paddingMode,
                       String provider) {
        this.algorithm = algorithm;
        this.provider = provider;
        this.cryptographicAlgorithm = algorithm + "/" + cipherMode + "/" + paddingMode;
    }

    private Cipher createCipher(int encryptDecryptMode,
                                byte[] keyValue,
                                byte[] initializationVector) throws Exception {
        SecretKeySpec keySpec = new SecretKeySpec(keyValue, algorithm);
        IvParameterSpec ivSpec = new IvParameterSpec(initializationVector);
        Cipher cipher = Cipher.getInstance(cryptographicAlgorithm, provider);
        cipher.init(encryptDecryptMode, keySpec, ivSpec);
        return cipher;
    }

    @Override
    public String toString() {
        return "CipherDebug{" +
                "provider=\"" + provider + '\"' +
                ", cryptographicAlgorithm=\"" + cryptographicAlgorithm + '\"' +
                '}';
    }

    private static String generateData(int length) {
        char[] chars = new char[length];
        Arrays.fill(chars, '0');
        return new String(chars);
    }

    public static void main(String[] args) throws Exception {
        Security.insertProviderAt(new BouncyCastleProvider(), 1);

        int numberOfChunks = 3;
        byte[] keyValue = Base64.getDecoder()
                .decode("yY7flqEdx95dojF/yY7flqEdx95dojF/".getBytes(StandardCharsets.UTF_8));
        byte[] initializationVector = "pjts4PzQIr9Pd2yb".getBytes(StandardCharsets.UTF_8);

        CipherDebug bouncyCastle = new CipherDebug("AES", "CBC", "PKCS5Padding", "BC");

        CipherDebug sunJCE = new CipherDebug("AES", "CBC", "PKCS5Padding", "SunJCE");

        Cipher bouncyCastleCipher = bouncyCastle.createCipher(Cipher.ENCRYPT_MODE,
                keyValue, initializationVector);

        Cipher sunJCECipher = sunJCE.createCipher(Cipher.ENCRYPT_MODE,
                keyValue, initializationVector);

        assert bouncyCastleCipher.getBlockSize() == sunJCECipher.getBlockSize();

        // blockSize = 16
        int blockSize = bouncyCastleCipher.getBlockSize();

        byte[] data = generateData(blockSize * numberOfChunks).getBytes(UTF_8);
        byte[] bouncyCastleUpdate = bouncyCastleCipher.update(data);
        byte[] sunJCEUpdate = sunJCECipher.update(data);

        //303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030
        System.out.println(printHexBinary(data));

        // CipherDebug{provider="BC", cryptographicAlgorithm="AES/CBC/PKCS5Padding"}
        // 1D4DE40480F0528D4F77E788817DA62902D98C9AE6DF9299F4F2D1836CC10924
        // 0320B10C8646D17E0755F8BBA1214ABF24D2E6E7F06184A78559793B23A9A341
        System.out.println(bouncyCastle.toString());
        System.out.println(printHexBinary(bouncyCastleUpdate));
        System.out.println(printHexBinary(bouncyCastleCipher.doFinal()));

        System.out.println();

        // CipherDebug{provider="SunJCE", cryptographicAlgorithm="AES/CBC/PKCS5Padding"}
        // 1D4DE40480F0528D4F77E788817DA62902D98C9AE6DF9299F4F2D1836CC109240320B10C8646D17E0755F8BBA1214ABF
        // 24D2E6E7F06184A78559793B23A9A341
        System.out.println(sunJCE.toString());
        System.out.println(printHexBinary(sunJCEUpdate));
        System.out.println(printHexBinary(sunJCECipher.doFinal()));

        // assertion fails
        assert Arrays.equals(bouncyCastleUpdate, sunJCEUpdate);
    }
}

The output:

// data
03030303030303030303030303030303030303030303030303030303030303030303030303030303030303030303030

// Bouncy Castle
CipherDebug{provider="BC", cryptographicAlgorithm="AES/CBC/PKCS5Padding"}
1D4DE40480F0528D4F77E788817DA62902D98C9AE6DF9299F4F2D1836CC10924
0320B10C8646D17E0755F8BBA1214ABF24D2E6E7F06184A78559793B23A9A341


// SunJCE
CipherDebug{provider="SunJCE", cryptographicAlgorithm="AES/CBC/PKCS5Padding"}
1D4DE40480F0528D4F77E788817DA62902D98C9AE6DF9299F4F2D1836CC109240320B10C8646D17E0755F8BBA1214ABF
24D2E6E7F06184A78559793B23A9A341

Answer 1

The extra data in the cipher text at the end is the padding, but you need to call Cipher.doFinal() in both cases - the cipher needs to know it has all the input data before it can add or remove padding.

Cipher.getIV() will return the IV. While the IV might be generated on encryption it is never part of the actual stream and is normally passed around as a parameter or generated.

In case it's the way that the output is getting chunked up that is causing confusion, there's no "standard" for this - in the case of BC it always holds onto a block until doFinal() arrives, for some ciphers the SunJCE provider doesn't, for HSMs the input may be buffered to first to make better use of the HSM, so a succession of updates may produce nothing, then suddenly a large chunk of processed data may appear. You need to rely on the return values from the updates and doFinals to tell handle the processed data correctly.