So I have a very basic openssl command that was provided to me openssl smime -encrypt -binary -aes-256-cbc -in $inPath -out $encryptedPath -outform DER $pubCert, this command also works correctly and outputs an encrypted file. I need to use the equivalent of this command in a java application, preferably without invoking process and using openssl itself (only because I feel like that is probably bad practice).
I have researched quite a lot and there does not seem to be any equivalent out there that I can find.. I have tried several things and most of them do not seem to work. The weird thing is... I am able to get a simple "Hello World" string to encrypt using the code I wrote (although I don't believe it was encrypting it correctly because I had the cipher set to "RSA" not "AES") but when the byte array was coming from a file, it silently failed and just wrote 0 bytes. Right now this is what my code looks like.
Cipher aes = Cipher.getInstance("RSA");
CertificateFactory certF = CertificateFactory.getInstance("X.509");
File public_cert = new File( getClass().getClassLoader().getResource("public.crt").getFile());
FileInputStream certIS = new FileInputStream(public_cert);
X509Certificate cert = (X509Certificate) certF.generateCertificate(certIS);
certIS.close();
aes.init(Cipher.ENCRYPT_MODE, cert);
File tarGz = new File("C:\\volatile\\generic.tar.gz");
FileInputStream fis = new FileInputStream(tarGz);
byte[] tarGzBytes = FileUtils.readFileToByteArray(tarGz);
tarGzBytes = "Hello World".getBytes();
ByteArrayInputStream bais = new ByteArrayInputStream("Hello World".getBytes());
File encFile = new File("C:\\volatile\\generic.tar.gz.enc");
FileOutputStream enc = new FileOutputStream(encFile);
CipherOutputStream cos = new CipherOutputStream(enc, aes);
cos.write(tarGzBytes);
//IOUtils.copy(fis, cos);
//IOUtils.copy(bais, cos);
cos.flush();
cos.close();
So this works, and encrypts a little file with Hello World encrypted in it. I don't believe this is AES-256-CBC though, and it does not work when I use the FileUtils.readFileToByteArray(tarGz), although the resulting byte array in a debugger is correctly sized at about 94MB. Which seems really odd to me, that it works with "Hello World".toByteArray() and not FileUtils.readAllBytes(tarGz). Also as a side note, the ByteArrayInputStream using IOUtils.copy works, whereas the FileInputStream version writes 0 bytes as well.
Also, when I set the cipher mode to AES/CBC/PKCS5Padding (because I found something online suggesting to set it to that and it looks more like what I want) I get the following error message:
java.security.InvalidKeyException: No installed provider supports this key: sun.security.rsa.RSAPublicKeyImpl
at javax.crypto.Cipher.chooseProvider(Cipher.java:892)
at javax.crypto.Cipher.init(Cipher.java:1724)
~~~~
If anyone has any suggestions, or if I need to provide more information please let me know. I am fairly stuck right now and I am at this point debating writing a script to simply run the openssl command and run that script from java...
After reading through @dave-thompson-085's answer I realized that there was a really good reason why I could not find what I was wanting to do. So therefore I decided to go ahead and just call the openssl process from java using a process builder. I was able to recreate the openssl command from above as a Process in java, start it and run it with the following code:
File cert = new File(getClass().getClassLoader().getResource("public.crt").getFile());
ProcessBuilder openSslBuilder = new ProcessBuilder("openssl", "smime", "-encrypt", "-binary",
"-aes-256-cbc", "-in", "C:\\volatile\\generic.tar.gz", "-out",
"C:\\volatile\\generic.tar.gz.enc", "-outform", "DER", cert.getPath());
Process openssl = openSslBuilder.start();
openssl.waitFor();
System.out.println(openssl.exitValue());
openssl.destroy();
Hopefully this helps someone else who is looking to attempt this as well and maybe save someone a bunch of time!
First, to be clear: the openssl smime command actually handles both S/MIME and CMS (aka PKCS7) formats; these are related but different standards that basically use different file formats for essentially the same cryptographic operations. With -outform DER you are actually doing CMS/PKCS7.
Second and more fundamental: CMS/PKCS7, and S/MIME, and most other common cryptographic schemes like PGP, actually does hybrid encryption. Your data is not actually encrypted with RSA; instead your data is encrypted with a symmetric algorithm (here AES-256-CBC, since you selected that) using a randomly generated key called the DEK (data encryption key) and the DEK is encrypted with RSA using the recipient's publickey (obtained from their certificate), and both of those results plus a good deal of metadata is arranged into a fairly complicated data structure. The recipient can parse the message to extract these pieces, then use RSA with their privatekey to decrypt the DEK, then AES-decrypt the data with the DEK. Note you always use RSA keys for RSA, and AES keys for AES; symmetric keys are pretty much all just bits and only vary in size, but public-key cryptographic keys including RSA (also DH, DSA, ECC and more) are much more complicated and cannot be intermixed.
Trying to encrypt data directly with RSA as you did, in addition to being wrong, won't work in general because RSA can only encrypt limited amounts of data, depending on the key size used, typically about 100-200 bytes. Symmetric encryption also has some limits, but they are generally much larger; AES-CBC is good for about 250,000,000,000,000,000 bytes.
If you want to implement this yourself, you need to read the standard for CMS particularly the section on EnvelopedData using KeyTransRecipientInfo (for RSA), combined with the rules for ASN.1 BER/DER encoding. This is not a simple job, although it can be done if you want to put the effort in.
If you can use a third-party library in Java, the 'bcpkix' jar from https://www.bouncycastle.org has routines that support CMS, among several other things. This is usually easy if you are writing a program to run yourself, or in your department. If this is to be delivered to outside users or customers who may not like having to manage a dependency, maybe not.
That said, running another program to do something isn't necessarily bad practice in my book, and can be done directly from java (no script). Unless you (need to) do it very often, such as 100 times a second.